Nov 01 20:01:44.850759 ip-10-0-142-223 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 20:01:45.229843 ip-10-0-142-223 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:01:45.229843 ip-10-0-142-223 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 20:01:45.229843 ip-10-0-142-223 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:01:45.229843 ip-10-0-142-223 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235415    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235506    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235510    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235513    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235516    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235519    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235522    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235524    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235527    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.235520 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235534    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235537    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235540    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235543    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235546    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235548    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235551    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235554    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235557    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235559    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235562    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235564    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235567    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235569    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235572    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235575    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235579    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235587    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235590    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.235830 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235592    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235595    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235597    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235600    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235602    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235604    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235614    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235618    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235621    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235623    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235626    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235629    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235631    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235634    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235637    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235640    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235643    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235645    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235648    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.236281 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235650    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235653    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235655    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235658    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235661    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235663    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235666    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235668    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235671    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235674    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235678    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235681    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235684    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235687    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235689    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235692    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235694    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235697    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235699    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.236747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235702    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235704    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235707    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235709    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235714    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235717    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235720    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235724    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235727    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235729    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235732    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235734    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235737    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235739    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235742    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235744    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235747    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235750    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235752    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.237192 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235754    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235757    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235759    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235762    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235764    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235767    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235770    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235772    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235775    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235777    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235781    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235783    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235785    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235788    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235790    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235793    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235795    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235798    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.235800    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236582    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.237697 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236591    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236596    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236600    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236605    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236608    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236612    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236615    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236617    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236620    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236623    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236626    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236628    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236631    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236633    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236636    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236638    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236641    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236644    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236646    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.238164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236649    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236652    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236655    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236657    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236660    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236663    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236665    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236668    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236670    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236672    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236675    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236679    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236682    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236686    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236690    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236692    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236696    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236698    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236701    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.238628 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236703    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236708    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236713    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236715    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236718    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236720    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236723    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236725    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236728    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236730    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236733    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236735    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236738    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236740    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236743    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236746    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236749    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236751    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.239088 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236754    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236757    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236760    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236762    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236765    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236767    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236770    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236773    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236776    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236778    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236781    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236783    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236788    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236791    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236793    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236795    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236798    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236800    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.239566 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236803    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236805    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236807    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236810    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236813    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236815    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236818    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236820    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236823    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236825    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236828    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236831    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236834    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236836    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236839    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236842    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236844    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236847    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236849    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236852    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.240004 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236855    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236857    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236860    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236864    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236867    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236870    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236872    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.236875    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237336    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237342    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237345    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237355    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237358    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237361    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237363    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237366    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237369    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237371    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237374    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.240463 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237377    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237385    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237387    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237390    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237393    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237396    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237398    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237401    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237403    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237408    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237410    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237413    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237416    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237419    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237421    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237424    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237426    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237429    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.240924 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237431    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237434    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237436    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237439    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237441    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237444    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237446    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237449    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237451    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237454    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237457    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237459    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237462    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237464    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237467    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237475    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237478    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237481    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237483    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237486    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.241378 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237488    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237504    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237506    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237509    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237511    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237514    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237516    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237519    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237522    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237524    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237527    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237529    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237532    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237535    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237540    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237542    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237545    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237547    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237550    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.241861 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237553    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237555    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237558    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237560    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237562    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237565    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237567    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237570    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237578    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237583    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237587    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237590    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237592    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237595    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237598    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237600    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237603    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237606    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237608    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237611    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.242349 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237613    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237615    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237620    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237624    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237626    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237628    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237631    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237636    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237638    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237640    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237643    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237646    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237648    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.237651    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238517    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238527    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238541    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238546    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238550    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 20:01:45.242836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238559    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238564    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238568    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238571    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238574    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238578    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238581    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238585    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238588    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238592    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238595    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238598    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238601    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238607    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238610    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238613    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238616    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238619    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238626    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238629    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238632    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238635    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238638    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238641    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 20:01:45.243516 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238644    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238647    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238650    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238654    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238657    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238660    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238663    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238666    2577 flags.go:64] FLAG: --enable-server="true"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238669    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238681    2577 flags.go:64] FLAG: --event-burst="100"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238685    2577 flags.go:64] FLAG: --event-qps="50"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238688    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238691    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238693    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238698    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238701    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238704    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238707    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238710    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238713    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238716    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238719    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238721    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238724    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238727    2577 flags.go:64] FLAG: --feature-gates=""
Nov 01 20:01:45.244103 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238731    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238736    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238739    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238742    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238745    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238748    2577 flags.go:64] FLAG: --help="false"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238751    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238754    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238757    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238760    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238763    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238767    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238769    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238772    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238775    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238778    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238781    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238789    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238792    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238795    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238798    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238801    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238804    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 20:01:45.244706 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238807    2577 flags.go:64] FLAG: --lock-file=""
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238810    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238812    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238815    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238821    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238824    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238827    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238830    2577 flags.go:64] FLAG: --logging-format="text"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238832    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238836    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238840    2577 flags.go:64] FLAG: --manifest-url=""
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238842    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238847    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238850    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238858    2577 flags.go:64] FLAG: --max-pods="110"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238861    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238864    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238866    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238869    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238872    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238875    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238878    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238889    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238892    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238895    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 20:01:45.245262 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238898    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238900    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238909    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238912    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238915    2577 flags.go:64] FLAG: --port="10250"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238918    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238923    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-084d411af08466a80"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238927    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238930    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238933    2577 flags.go:64] FLAG: --register-node="true"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238936    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238940    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238943    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238946    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238949    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238953    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238956    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238966    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238970    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238974    2577 flags.go:64] FLAG: --runonce="false"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238977    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238980    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238983    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238986    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238988    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238992    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 20:01:45.245938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238995    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.238998    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239000    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239003    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239006    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239009    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239012    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239015    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239017    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239028    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239031    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239034    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239042    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239045    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239048    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239050    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239053    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239056    2577 flags.go:64] FLAG: --v="2"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239060    2577 flags.go:64] FLAG: --version="false"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239065    2577 flags.go:64] FLAG: --vmodule=""
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239069    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239072    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239236    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239243    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.246907 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239246    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239249    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239252    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239254    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239257    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239259    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239262    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239265    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239267    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239271    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239275    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239279    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239282    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239284    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239288    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239292    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239296    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239299    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239302    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.247478 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239304    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239308    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239311    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239314    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239316    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239319    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239322    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239324    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239327    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239329    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239332    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239334    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239338    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239341    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239343    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239346    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239348    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239351    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239353    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239356    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.247941 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239359    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239361    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239364    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239367    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239369    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239372    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239374    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239377    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239379    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239382    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239385    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239387    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239403    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239407    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239409    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239412    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239414    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239417    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239419    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.248400 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239422    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239425    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239427    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239429    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239432    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239436    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239438    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239441    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239444    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239446    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239449    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239452    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239454    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239457    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239459    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239462    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239464    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239467    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239470    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239472    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.248867 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239475    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239477    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239480    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239482    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239485    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239487    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239508    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239511    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239514    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239516    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239519    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239522    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239524    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239527    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239529    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239532    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239535    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239538    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239541    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239543    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.249339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239546    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239548    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239551    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239553    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239556    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239558    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.239585    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239623    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239626    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239628    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239633    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239637    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239640    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239643    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.249950 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239645    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239648    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239651    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239655    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239663    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239666    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239669    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239672    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239674    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239677    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239679    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239681    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239684    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239687    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239689    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239692    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239695    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239697    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239700    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239702    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.250313 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239705    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239707    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239710    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239713    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239715    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239717    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239720    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239723    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239725    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239728    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239730    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239733    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239735    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239738    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239740    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239744    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239746    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239754    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239757    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239759    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.250792 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239762    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239764    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239767    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239769    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239771    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239774    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239776    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239779    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239782    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239784    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239787    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239789    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239792    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239794    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239797    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239800    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239802    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239805    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239807    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239810    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.251257 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239812    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239815    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239817    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239820    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239822    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239825    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239827    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239831    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239833    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239836    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239844    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239847    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239849    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239852    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239854    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239857    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239859    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239862    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239864    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239867    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.251770 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239869    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239872    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239874    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239877    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239879    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239881    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239884    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239886    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239889    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239891    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239894    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239897    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239899    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239901    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239904    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239906    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239909    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.239913    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.248003    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 20:01:45.252231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.248020    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.248691    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.248698    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248760    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248765    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248769    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248772    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248775    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248778    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248781    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248784    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248787    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248789    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248792    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248794    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248797    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248800    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248802    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248805    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.252701 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248808    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248810    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248813    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248815    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248818    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248820    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248828    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248831    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248836    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248839    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248842    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248845    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248849    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248853    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248856    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248859    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248862    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248865    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.253138 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248867    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248870    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248873    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248876    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248879    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248882    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248884    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248887    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248890    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248893    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248895    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248901    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248904    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248907    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248910    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248913    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248915    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248918    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.253725 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248922    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248925    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248929    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248932    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248935    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248938    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248942    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248944    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248947    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248950    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248952    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248955    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248958    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248960    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248963    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248966    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248968    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248971    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248974    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.254164 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248976    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248979    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248982    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248984    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248988    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248990    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248993    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248996    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.248999    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249001    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249004    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249006    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249009    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249012    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249014    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249017    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249020    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249023    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249025    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249028    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.254629 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249030    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249033    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249035    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249038    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249041    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249043    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249046    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249048    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249051    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249053    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249056    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249058    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249061    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.249089    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249236    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249241    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.255101 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249244    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249247    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249250    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249253    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249256    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249259    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249261    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249264    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249267    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249269    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249272    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249275    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249279    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249282    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249285    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249288    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249291    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249294    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249296    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.255485 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249299    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249302    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249304    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249307    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249310    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249312    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249315    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249317    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249320    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249322    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249325    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249327    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249330    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249332    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249335    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249337    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249340    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249344    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249350    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.255956 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249352    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249355    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249358    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249363    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249366    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249370    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249373    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249376    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249379    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249382    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249384    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249387    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249389    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249392    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249394    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249397    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249400    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.256411 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249405    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249408    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249410    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249413    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249415    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249418    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249420    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249423    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249425    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249428    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249430    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249434    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249437    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249439    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249442    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249444    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249449    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249452    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249454    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.256865 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249457    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249460    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249462    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249465    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249467    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249470    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249472    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249475    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249478    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249480    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249482    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249485    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249487    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249504    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249508    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249510    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249513    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249515    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249518    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.257330 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249520    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249523    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249525    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249528    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249531    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249534    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249536    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.249560    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249593    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249597    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249600    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249603    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249606    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249609    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249612    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:01:45.257771 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249614    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249617    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249620    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249622    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249625    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249628    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249630    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249632    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249635    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249638    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249640    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249643    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249645    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249648    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249650    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249653    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249655    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249658    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249660    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:01:45.258139 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249663    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249665    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249668    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249671    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249674    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249677    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249679    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249682    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249684    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249687    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249689    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249692    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249694    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249697    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249700    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249704    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249707    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249709    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249712    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:01:45.258596 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249714    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249717    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249719    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249725    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249728    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249730    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249733    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249735    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249738    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249741    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249743    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249746    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249748    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249751    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249753    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249756    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249758    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249761    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249764    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249767    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:01:45.259037 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249769    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249772    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249774    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249777    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249780    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249782    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249803    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249806    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249810    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249814    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249817    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249820    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249822    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249825    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249827    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249831    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249834    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249836    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249839    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:01:45.259549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249841    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249844    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249847    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249849    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249852    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249859    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249861    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249864    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249866    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249869    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249871    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249876    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249879    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249881    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249884    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249886    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249889    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249891    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249894    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:01:45.259982 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249897    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:45.249899    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.249952    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.252279    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.253281    2577 server.go:1014] "Starting client certificate rotation"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.253380    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.254334    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.255407    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:01:45.260437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.256596    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:01:45.270554 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.270537    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 01 20:01:45.274383 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.274370    2577 log.go:25] "Validated CRI v1 image API"
Nov 01 20:01:45.275203 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.275180    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 20:01:45.280159 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.280140    2577 fs.go:135] Filesystem UUIDs: map[65b92ded-d54c-42af-b003-e9ff6d74fe72:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89243e4c-36cf-4a76-82d6-680cc6d0c62f:/dev/nvme0n1p4]
Nov 01 20:01:45.280264 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.280158    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 20:01:45.285621 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.285485    2577 manager.go:217] Machine: {Timestamp:2026-11-01 20:01:45.284325967 +0000 UTC m=+0.329463171 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3254411 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d637e638f704c2b2d4e9ba568b24 SystemUUID:ec27d637-e638-f704-c2b2-d4e9ba568b24 BootID:dc9b979d-c3c8-4bcf-9628-7b12a15aa898 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:2b:4c:93:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:2b:4c:93:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:86:6d:cb:fd:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 20:01:45.285621 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.285620    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 20:01:45.285734 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.285695    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 20:01:45.287537 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287510    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 20:01:45.287662 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287540    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-223.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 20:01:45.287701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287665    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 20:01:45.287701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287673    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 20:01:45.287701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287698    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:01:45.287782 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.287715    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:01:45.289023 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.289013    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 20:01:45.293275 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.293264    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 20:01:45.293387 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.293378    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 20:01:45.294304 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.294294    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 20:01:45.294346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.294308    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 20:01:45.294346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.294317    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 20:01:45.294346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.294332    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 01 20:01:45.294346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.294341    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 20:01:45.297460 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.297441    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:01:45.299835 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.299821    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 20:01:45.306583 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.306570    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:01:45.307538 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307524    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 20:01:45.307600 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307563    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 20:01:45.307744 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307732    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 20:01:45.307803 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307750    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 20:01:45.307803 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307759    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 20:01:45.307803 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307769    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 20:01:45.307803 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307779    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 20:01:45.307803 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.307787    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 20:01:45.308622 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308611    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 20:01:45.308679 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308625    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 20:01:45.308679 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308639    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 20:01:45.308679 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308648    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 20:01:45.308679 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308668    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 20:01:45.308812 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.308681    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 20:01:45.309430 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.309421    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 20:01:45.309484 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.309435    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 20:01:45.313109 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.313094    2577 server.go:1263] "Started kubelet"
Nov 01 20:01:45.313239 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.313172    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 20:01:45.313284 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.313213    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 20:01:45.313332 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.313298    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 20:01:45.313868 ip-10-0-142-223 systemd[1]: Started Kubernetes Kubelet.
Nov 01 20:01:45.314598 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.314583    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 20:01:45.318368 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.318354    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 20:01:45.318428 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.318385    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 20:01:45.323125 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.323106    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 20:01:45.323791 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.323776    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.323847    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324551    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324551    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324576    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324588    2577 factory.go:55] Registering systemd factory
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324623    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324695    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324702    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324806    2577 factory.go:153] Registering CRI-O factory
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324822    2577 factory.go:223] Registration of the crio container factory successfully
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324880    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324898    2577 factory.go:103] Registering Raw factory
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.324913    2577 manager.go:1199] Started watching for new ooms in manager
Nov 01 20:01:45.326106 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.325840    2577 manager.go:319] Starting recovery of all containers
Nov 01 20:01:45.338632 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.338612    2577 manager.go:324] Recovery completed
Nov 01 20:01:45.344402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.344386    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.346131 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346104    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.346219 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346149    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.346219 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346165    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.346770 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346749    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 01 20:01:45.346770 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346759    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.346880 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346760    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 20:01:45.346880 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.346792    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 20:01:45.347502 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.347473    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.347572 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.347515    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.347572 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.347531    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.348888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.348875    2577 policy_none.go:50] "Start"
Nov 01 20:01:45.348888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.348890    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 20:01:45.348980 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.348900    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 20:01:45.350837 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.350821    2577 policy_none.go:44] "Start"
Nov 01 20:01:45.401069 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.401037    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 20:01:45.406267 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406253    2577 manager.go:341] "Starting Device Plugin manager"
Nov 01 20:01:45.406338 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.406278    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 20:01:45.406338 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406286    2577 server.go:87] "Starting device plugin registration server"
Nov 01 20:01:45.406557 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406546    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 20:01:45.406601 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406560    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 20:01:45.406672 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406659    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 20:01:45.406748 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406738    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 20:01:45.406748 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.406748    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 20:01:45.407472 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.407455    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 20:01:45.407554 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.407520    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:01:45.421596 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.421469    2577 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2612 into the system container \"/system.slice\": <nil>"
Nov 01 20:01:45.456451 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.456356    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-223.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 20:01:45.457297 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.456377    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-223.us-west-1.compute.internal.18e40452cf5c663b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-223.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-223.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:01:45.313068603 +0000 UTC m=+0.358205807,LastTimestamp:2026-11-01 20:01:45.313068603 +0000 UTC m=+0.358205807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-223.us-west-1.compute.internal,}"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.505655    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-223.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.506661    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.507527    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.507555    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.507565    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.520888 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.507604    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.530444 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.530426    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 20:01:45.530543 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.530464    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m6tjz"
Nov 01 20:01:45.548646 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.548624    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 20:01:45.549719 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.549705    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 20:01:45.549719 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.549722    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 20:01:45.549859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.549739    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 20:01:45.549859 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.549766    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 20:01:45.614454 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.614433    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:45.616195 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.616175    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:45.650450 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.650409    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"]
Nov 01 20:01:45.650546 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.650512    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.651358 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.651343    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.651421 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.651371    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.651421 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.651386    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.653357 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.653345    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.653532 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.653517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.653571 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.653548    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.654204 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654192    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.654258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654194    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.654258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654221    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.654258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654236    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.654258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654238    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.654258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.654252    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.656117 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.656101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.656209 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.656128    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:01:45.656843 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.656826    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:01:45.656925 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.656854    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:01:45.656925 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.656869    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:01:45.689599 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.689583    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.691577 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.691562    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.713478 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.713454    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:45.714586 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.714573    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:45.715134 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.715119    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.715195 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:45.715140    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-223.us-west-1.compute.internal\": node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:01:45.724572 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.724553    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.727794 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.727779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.727866 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.727801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.727866 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.727817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.827965 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.827908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.827965 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.827935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.828112 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.827982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.828112 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.827994    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.828112 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.828005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.828112 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.828020    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.835096 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.835079    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:01:45.835703 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.835692    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.938571 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.938553    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:01:45.992280 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.992257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:45.994637 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:45.994624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:01:46.252859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.252830    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 20:01:46.253421 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.252948    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:01:46.253421 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.253006    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:01:46.253421 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.253014    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 20:01:46.253421 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.253041    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-223.us-west-1.compute.internal.18e40452d1557f09\": read tcp 10.0.142.223:42354->3.138.241.54:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-223.us-west-1.compute.internal.18e40452d1557f09  default   4860 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-223.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-223.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-223.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:01:45 +0000 UTC,LastTimestamp:2026-11-01 20:01:45.507569152 +0000 UTC m=+0.552706357,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-223.us-west-1.compute.internal,}"
Nov 01 20:01:46.298474 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.298443    2577 apiserver.go:52] "Watching apiserver"
Nov 01 20:01:46.324237 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.324216    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 20:01:46.534235 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.534143    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:56:46 +0000 UTC" deadline="2028-07-19 18:17:45.754405992 +0000 UTC"
Nov 01 20:01:46.534235 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.534191    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15022h15m59.22021904s"
Nov 01 20:01:46.588975 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.588932    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 20:01:46.589296 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.589278    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-q4r47","openshift-network-operator/iptables-alerter-sjn27","kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal","openshift-image-registry/node-ca-zq4jc","openshift-multus/network-metrics-daemon-hqhjj","openshift-ovn-kubernetes/ovnkube-node-sh57k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v","openshift-multus/multus-additional-cni-plugins-wrz6h","kube-system/global-pull-secret-syncer-nn89g","openshift-cluster-node-tuning-operator/tuned-dvx8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal","kube-system/konnectivity-agent-ktldr","openshift-dns/node-resolver-fm8ml","openshift-multus/multus-cmcvj"]
Nov 01 20:01:46.591847 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.591829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:46.591921 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.591903    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:46.593831 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.593817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.595849 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.595830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.597989 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.597969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.598062 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.598030    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:46.600863 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.600390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.602927 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.602907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.605241 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.605226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.607470 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.607452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.607583 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.607547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:46.609845 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.609826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.612198 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.612180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.614514 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.614499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.617028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.617014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.625672 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.625654    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 20:01:46.634533 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.634616 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.634616 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.634616 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.634756 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.634756 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rj4\" (UniqueName: \"kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.634756 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.634756 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.634756 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42mm\" (UniqueName: \"kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.634906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.634996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9nv\" (UniqueName: \"kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635087 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tcn\" (UniqueName: \"kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635452 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmk97\" (UniqueName: \"kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.635922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6bb\" (UniqueName: \"kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.635990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgx5\" (UniqueName: \"kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.636231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsg4k\" (UniqueName: \"kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.636642 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.636963 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwb4\" (UniqueName: \"kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.636963 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.636963 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.636694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.651186 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.651167    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\""
Nov 01 20:01:46.651186 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.651185    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.651320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.651242    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 20:01:46.651320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.651242    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.652608 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.652591    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.652679 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.652664    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 20:01:46.652724 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.652664    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.652724 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.652672    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\""
Nov 01 20:01:46.657067 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657053    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.657336 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657324    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\""
Nov 01 20:01:46.657646 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657632    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.657720 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657675    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 20:01:46.657893 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657878    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 20:01:46.657950 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.657897    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 20:01:46.658306 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.658294    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 20:01:46.659257 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.659245    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.659532 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.659520    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\""
Nov 01 20:01:46.706067 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.706052    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 20:01:46.706144 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.706135    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\""
Nov 01 20:01:46.706144 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.706137    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.706243 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.706235    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 20:01:46.706312 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.706294    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\""
Nov 01 20:01:46.707477 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.707460    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 20:01:46.707551 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.707494    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\""
Nov 01 20:01:46.707551 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.707541    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.707650 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.707557    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.707650 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.707597    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 20:01:46.712092 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712076    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.712178 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712170    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 20:01:46.712234 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712170    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 20:01:46.712234 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712225    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 20:01:46.712363 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712350    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.712942 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712928    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 20:01:46.713020 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.712950    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 20:01:46.713086 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.713046    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\""
Nov 01 20:01:46.713141 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.713110    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\""
Nov 01 20:01:46.713206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.713158    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 20:01:46.713272 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.713214    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6rmv"
Nov 01 20:01:46.736974 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.736953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737047 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.736993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737047 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.737114 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737114 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737114 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737088    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737114 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rj4\" (UniqueName: \"kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42mm\" (UniqueName: \"kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.737285 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737387    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.737574    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9nv\" (UniqueName: \"kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.737633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.737647    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:47.237627463 +0000 UTC m=+2.282764681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737674    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tcn\" (UniqueName: \"kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737923    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737960    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.737999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.738402 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738217    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738252    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738286    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.739206 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.738349    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.738572    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:01:47.238558701 +0000 UTC m=+2.283695902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738623    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738530    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738665    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740028 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738788    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk97\" (UniqueName: \"kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6bb\" (UniqueName: \"kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738950    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738978    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.738953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.740900 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgx5\" (UniqueName: \"kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739242    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739386    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739155    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739435    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.741826 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739858    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739955    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.742479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.739978    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg4k\" (UniqueName: \"kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwb4\" (UniqueName: \"kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740599    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740658    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.740810    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.742203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.742275    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.743013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.742313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.743498 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.742346    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.756954 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.756936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:46.800432 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.800225    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:01:46.800432 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.800251    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:01:46.800432 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.800265    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:46.800432 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:46.800322    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:47.300304481 +0000 UTC m=+2.345441688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:46.800887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.800798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tcn\" (UniqueName: \"kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:46.800887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.800829    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rj4\" (UniqueName: \"kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:46.801039 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.801017    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9nv\" (UniqueName: \"kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:46.801093 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.801052    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6bb\" (UniqueName: \"kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.801550 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.801510    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk97\" (UniqueName: \"kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.802179 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.802160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42mm\" (UniqueName: \"kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:46.802962 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.802941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg4k\" (UniqueName: \"kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:46.803027 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.802941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgx5\" (UniqueName: \"kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:46.803027 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.803004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwb4\" (UniqueName: \"kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:46.931439 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.931402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:01:46.937658 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.937634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:01:46.993410 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:46.993381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:01:47.004922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.004905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:01:47.032651 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.032634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:01:47.034792 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.034770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:01:47.074701 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.074649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:01:47.079580 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.079550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmcvj"
Nov 01 20:01:47.244272 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.244245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:47.244377 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.244345    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:47.244432 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.244416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:01:48.244401504 +0000 UTC m=+3.289538710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:47.244485 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.244434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:47.244541 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.244498    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:47.244588 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.244543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:48.244519498 +0000 UTC m=+3.289656696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:47.254751 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.254725    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bc8eba7840edd90e06bc583d329627.slice/crio-4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002 WatchSource:0}: Error finding container 4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002: Status 404 returned error can't find the container with id 4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002
Nov 01 20:01:47.255414 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.255398    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d138da692bc0e2c453d9d2e3886bfc.slice/crio-4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8 WatchSource:0}: Error finding container 4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8: Status 404 returned error can't find the container with id 4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8
Nov 01 20:01:47.260572 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.260553    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:01:47.262260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.262240    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:47.345449 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.345374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:47.345604 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.345503    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:01:47.345604 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.345540    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:01:47.345604 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.345552    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:47.345604 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:47.345605    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:48.345584513 +0000 UTC m=+3.390721711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:47.353320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.353295    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:47.552518 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.552479    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:01:47.553833 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.553783    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerStarted","Data":"4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002"}
Nov 01 20:01:47.554796 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.554771    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"88d138da692bc0e2c453d9d2e3886bfc","Type":"ContainerStarted","Data":"4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8"}
Nov 01 20:01:47.660031 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.659999    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8398821f_a5b8_4ddf_a46f_e1123e5bfbfc.slice/crio-134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c WatchSource:0}: Error finding container 134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c: Status 404 returned error can't find the container with id 134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c
Nov 01 20:01:47.716801 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.716743    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:47 +0000 UTC" deadline="2028-07-09 17:10:51.514860902 +0000 UTC"
Nov 01 20:01:47.716801 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:47.716796    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14781h9m3.798067484s"
Nov 01 20:01:47.771448 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.771423    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7319d0_ad0f_4af5_b0b6_31664beab34e.slice/crio-4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94 WatchSource:0}: Error finding container 4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94: Status 404 returned error can't find the container with id 4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94
Nov 01 20:01:47.821774 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.821745    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19535b6d_3a55_4cd3_b3f5_d0f8331fa8dc.slice/crio-9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1 WatchSource:0}: Error finding container 9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1: Status 404 returned error can't find the container with id 9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1
Nov 01 20:01:47.917747 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.917710    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d9db4b_329f_4358_b755_ae66f4260444.slice/crio-a2b6c7126b209ddfe100d9f572f6a83aa7965fcce6c743ed11cee85a584addaa WatchSource:0}: Error finding container a2b6c7126b209ddfe100d9f572f6a83aa7965fcce6c743ed11cee85a584addaa: Status 404 returned error can't find the container with id a2b6c7126b209ddfe100d9f572f6a83aa7965fcce6c743ed11cee85a584addaa
Nov 01 20:01:47.921002 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.920960    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ff9c3f_675e_4693_adb0_9bdf4308347c.slice/crio-ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a WatchSource:0}: Error finding container ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a: Status 404 returned error can't find the container with id ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a
Nov 01 20:01:47.921911 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.921875    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c09b58_e615_4304_b62a_a9733d309f60.slice/crio-ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b WatchSource:0}: Error finding container ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b: Status 404 returned error can't find the container with id ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b
Nov 01 20:01:47.970908 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.970885    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0391dc_0199_4138_97d2_4db3371dda0f.slice/crio-49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2 WatchSource:0}: Error finding container 49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2: Status 404 returned error can't find the container with id 49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2
Nov 01 20:01:47.971636 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:47.971609    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adb9ea9_0207_415d_91c4_d85a59b4746c.slice/crio-f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6 WatchSource:0}: Error finding container f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6: Status 404 returned error can't find the container with id f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6
Nov 01 20:01:48.022228 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:01:48.022206    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067dc6af_f795_489f_820d_b81b63250a86.slice/crio-a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5 WatchSource:0}: Error finding container a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5: Status 404 returned error can't find the container with id a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5
Nov 01 20:01:48.254912 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.254822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:48.254912 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.254865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:48.255464 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.254976    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:48.255464 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.255041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:01:50.255022245 +0000 UTC m=+5.300159454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:48.255464 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.254976    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:48.255464 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.255099    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:50.255086301 +0000 UTC m=+5.300223499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:48.355812 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.355777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:48.355974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.355935    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:01:48.355974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.355956    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:01:48.355974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.355974    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:48.356097 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.356031    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:50.356015414 +0000 UTC m=+5.401152608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:48.550812 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.550731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:48.550812 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.550754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:48.551038 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.550849    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:48.551038 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.550877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:48.551038 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.550948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:48.551200 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:48.551029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:48.558968 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.558937    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerStarted","Data":"f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6"}
Nov 01 20:01:48.560224 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.560172    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvx8k" event={"ID":"99d9db4b-329f-4358-b755-ae66f4260444","Type":"ContainerStarted","Data":"a2b6c7126b209ddfe100d9f572f6a83aa7965fcce6c743ed11cee85a584addaa"}
Nov 01 20:01:48.561692 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.561653    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sjn27" event={"ID":"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc","Type":"ContainerStarted","Data":"9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1"}
Nov 01 20:01:48.562991 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.562958    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zq4jc" event={"ID":"cb7319d0-ad0f-4af5-b0b6-31664beab34e","Type":"ContainerStarted","Data":"4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94"}
Nov 01 20:01:48.564472 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.564445    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fm8ml" event={"ID":"067dc6af-f795-489f-820d-b81b63250a86","Type":"ContainerStarted","Data":"a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5"}
Nov 01 20:01:48.565681 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.565657    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmcvj" event={"ID":"0a0391dc-0199-4138-97d2-4db3371dda0f","Type":"ContainerStarted","Data":"49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2"}
Nov 01 20:01:48.570792 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.567431    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b"}
Nov 01 20:01:48.573901 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.573873    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v" event={"ID":"b5ff9c3f-675e-4693-adb0-9bdf4308347c","Type":"ContainerStarted","Data":"ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a"}
Nov 01 20:01:48.575267 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.575237    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktldr" event={"ID":"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc","Type":"ContainerStarted","Data":"134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c"}
Nov 01 20:01:48.717462 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.717406    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:47 +0000 UTC" deadline="2028-06-07 02:14:54.497250578 +0000 UTC"
Nov 01 20:01:48.717462 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:48.717459    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13998h13m5.779795499s"
Nov 01 20:01:50.270486 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.270454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:50.270974 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.270566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:50.270974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.270666    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:50.270974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.270730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:54.270710061 +0000 UTC m=+9.315847270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:50.270974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.270751    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:50.270974 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.270779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:01:54.27076857 +0000 UTC m=+9.315905770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:50.371794 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.371714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:50.372008 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.371972    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:01:50.372008 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.371995    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:01:50.372008 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.372006    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:50.372169 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.372130    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:54.372110633 +0000 UTC m=+9.417247840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.550946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.551093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.551176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.551277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:50.551347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:50.553177 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:50.551460    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.550523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.550523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:52.550697    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:52.550780    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.550535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:52.551220 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:52.550891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:52.613260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.613225    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"88d138da692bc0e2c453d9d2e3886bfc","Type":"ContainerStarted","Data":"d7ab2ccd5f2644be416c688e4b58f50750a2bfd67497ae00ac4cf76cc9cad0ea"}
Nov 01 20:01:52.616542 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.615751    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="291585c6b2951cb36cef8d8b1391e6bbe6c2f0c2d457ba48fe5634ed0b28f86d" exitCode=0
Nov 01 20:01:52.616542 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.615840    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"291585c6b2951cb36cef8d8b1391e6bbe6c2f0c2d457ba48fe5634ed0b28f86d"}
Nov 01 20:01:52.620079 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.620053    2577 generic.go:356] "Generic (PLEG): container finished" podID="61bc8eba7840edd90e06bc583d329627" containerID="edae60eccbe3c39f4ad5cf478257de7381265fdf58ce08dbaeb7bdc2d8a00d40" exitCode=0
Nov 01 20:01:52.620176 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.620096    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerDied","Data":"edae60eccbe3c39f4ad5cf478257de7381265fdf58ce08dbaeb7bdc2d8a00d40"}
Nov 01 20:01:52.866155 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:52.866077    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal" podStartSLOduration=6.866054246 podStartE2EDuration="6.866054246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:52.740687504 +0000 UTC m=+7.785824721" watchObservedRunningTime="2026-11-01 20:01:52.866054246 +0000 UTC m=+7.911191463"
Nov 01 20:01:53.627930 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:53.627889    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktldr" event={"ID":"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc","Type":"ContainerStarted","Data":"6e85ef6078154b1324dd214b6272979511c6c498617bc6ede9408e6dc36633fa"}
Nov 01 20:01:53.633506 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:53.633474    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerStarted","Data":"820c98c52d27c6d0dcfcac309dc7aad1a6f7d079757d857e5f50ad3b67416c13"}
Nov 01 20:01:53.757634 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:53.757559    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ktldr" podStartSLOduration=2.298401006 podStartE2EDuration="7.757538589s" totalImagesPullingTime="5.459137583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:53.7572115 +0000 UTC m=+8.802348716" watchObservedRunningTime="2026-11-01 20:01:53.757538589 +0000 UTC m=+8.802675807"
Nov 01 20:01:54.307169 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.307072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:54.307336 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.307196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:54.307404 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.307365    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:54.307457 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.307427    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.307408749 +0000 UTC m=+17.352545955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:01:54.307523 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.307497    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:54.307601 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.307532    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.307520607 +0000 UTC m=+17.352657813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:01:54.408361 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.408322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:54.408554 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.408514    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:01:54.408639 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.408559    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:01:54.408639 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.408573    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:54.408735 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.408638    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:02.408618313 +0000 UTC m=+17.453755524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:01:54.550505 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.550446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:54.550683 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.550621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:54.551079 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.551056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:54.551184 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.551158    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:54.551253 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.551224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:54.551486 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:54.551464    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:54.641587 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.639721    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zq4jc" event={"ID":"cb7319d0-ad0f-4af5-b0b6-31664beab34e","Type":"ContainerStarted","Data":"5716d2cf9063d4849bc225d999be90af42307288e3327e8e8b5b59254bfa7bdc"}
Nov 01 20:01:54.763471 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.763400    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" podStartSLOduration=8.763380784 podStartE2EDuration="8.763380784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:53.883299842 +0000 UTC m=+8.928437060" watchObservedRunningTime="2026-11-01 20:01:54.763380784 +0000 UTC m=+9.808518003"
Nov 01 20:01:54.763906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:54.763871    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zq4jc" podStartSLOduration=2.753159438 podStartE2EDuration="8.763859101s" totalImagesPullingTime="6.010699663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:54.762879488 +0000 UTC m=+9.808016706" watchObservedRunningTime="2026-11-01 20:01:54.763859101 +0000 UTC m=+9.808996321"
Nov 01 20:01:55.644414 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:55.644377    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v" event={"ID":"b5ff9c3f-675e-4693-adb0-9bdf4308347c","Type":"ContainerStarted","Data":"808524a79a82546aac1b6fa058a18011a11f0b8273060c3e3a1e5409debfdcc9"}
Nov 01 20:01:56.550533 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:56.550486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:56.550737 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:56.550651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:56.551109 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:56.551091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:56.551196 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:56.551171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:56.551267 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:56.551238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:56.551318 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:56.551306    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:57.050525 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.050499    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 20:01:57.124388 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.124361    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:57.124956 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.124935    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:01:57.415283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.414945    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:01:57.050520622Z","UUID":"d2adbe38-0f55-412f-b212-2f63b113a85b","Handler":null,"Name":"","Endpoint":""}
Nov 01 20:01:57.417224 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.417203    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 20:01:57.417351 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.417235    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 20:01:57.651303 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.651215    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v" event={"ID":"b5ff9c3f-675e-4693-adb0-9bdf4308347c","Type":"ContainerStarted","Data":"8d2b4a4aa98b63686a84b0f17a93e74e739296a2162b5d75a31c7752a20e424f"}
Nov 01 20:01:57.654645 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.654179    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerStarted","Data":"77be49e885e62f9bf5ee8467145e99cb104afe7315741ce9608fcbc7ed9a783e"}
Nov 01 20:01:57.655882 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.655852    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fm8ml" event={"ID":"067dc6af-f795-489f-820d-b81b63250a86","Type":"ContainerStarted","Data":"37b54cc6e8ab77c1a4368350c289027f6b279e1736eab6272d868c8222af976c"}
Nov 01 20:01:57.657674 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.657650    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmcvj" event={"ID":"0a0391dc-0199-4138-97d2-4db3371dda0f","Type":"ContainerStarted","Data":"1e393bfbc171de3e4c297b16cfd4501b5eccf3c53ac1d4119e83edbdaed3f18d"}
Nov 01 20:01:57.907988 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:57.907918    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fm8ml" podStartSLOduration=2.828142658 podStartE2EDuration="11.907900562s" totalImagesPullingTime="9.079757904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:57.907128053 +0000 UTC m=+12.952265270" watchObservedRunningTime="2026-11-01 20:01:57.907900562 +0000 UTC m=+12.953037779"
Nov 01 20:01:58.026194 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.026108    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cmcvj" podStartSLOduration=3.323788222 podStartE2EDuration="12.026093218s" totalImagesPullingTime="8.702304996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:58.025191086 +0000 UTC m=+13.070328339" watchObservedRunningTime="2026-11-01 20:01:58.026093218 +0000 UTC m=+13.071230428"
Nov 01 20:01:58.550136 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.550102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:01:58.550618 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:58.550225    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:01:58.550618 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.550599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:01:58.550720 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:58.550701    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:01:58.550764 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.550598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:01:58.550814 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:01:58.550779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:01:58.662755 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.662717    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="77be49e885e62f9bf5ee8467145e99cb104afe7315741ce9608fcbc7ed9a783e" exitCode=0
Nov 01 20:01:58.662906 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.662810    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"77be49e885e62f9bf5ee8467145e99cb104afe7315741ce9608fcbc7ed9a783e"}
Nov 01 20:01:58.667844 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.667816    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sjn27" event={"ID":"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc","Type":"ContainerStarted","Data":"a1998f1db3ef85563343b464add9e9141c829adc67c429aac5ff2827814460b0"}
Nov 01 20:01:58.896568 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:58.896460    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sjn27" podStartSLOduration=3.976913198 podStartE2EDuration="12.896445211s" totalImagesPullingTime="8.919532013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:58.896081005 +0000 UTC m=+13.941218220" watchObservedRunningTime="2026-11-01 20:01:58.896445211 +0000 UTC m=+13.941582423"
Nov 01 20:01:59.671232 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:59.671190    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v" event={"ID":"b5ff9c3f-675e-4693-adb0-9bdf4308347c","Type":"ContainerStarted","Data":"263f110ea5c7da4ddb6974955a015cc497a8eba38e5766497b22dab787f5e446"}
Nov 01 20:01:59.673647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:59.673616    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvx8k" event={"ID":"99d9db4b-329f-4358-b755-ae66f4260444","Type":"ContainerStarted","Data":"69bc81e9ada2cef2b6d494cbd3fc7c807685a35529019f356cdfe42aef7f36bc"}
Nov 01 20:01:59.790732 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:59.790674    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v" podStartSLOduration=3.515855294 podStartE2EDuration="13.790660437s" totalImagesPullingTime="10.274805143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:59.789770451 +0000 UTC m=+14.834907687" watchObservedRunningTime="2026-11-01 20:01:59.790660437 +0000 UTC m=+14.835797661"
Nov 01 20:01:59.906656 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:01:59.906601    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dvx8k" podStartSLOduration=2.911046565 podStartE2EDuration="13.906583651s" totalImagesPullingTime="10.995537086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:59.906033512 +0000 UTC m=+14.951170724" watchObservedRunningTime="2026-11-01 20:01:59.906583651 +0000 UTC m=+14.951720866"
Nov 01 20:02:00.550236 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:00.550201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:00.550408 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:00.550251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:00.550408 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:00.550348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:00.550408 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:00.550357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:00.550582 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:00.550437    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:00.550582 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:00.550534    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:00.678482 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:00.678282    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="a3119840144feba6cbe89f0eaeeea199ca9e39dab33285607190477ba7ce2550" exitCode=0
Nov 01 20:02:00.678938 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:00.678339    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"a3119840144feba6cbe89f0eaeeea199ca9e39dab33285607190477ba7ce2550"}
Nov 01 20:02:02.377482 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.377455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:02.377891 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.377517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:02.377891 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.377618    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:02:02.377891 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.377640    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 20:02:02.377891 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.377684    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:18.377668143 +0000 UTC m=+33.422805354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 20:02:02.377891 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.377700    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret podName:2a563d52-ad66-4e70-98e9-10266df43ddd nodeName:}" failed. No retries permitted until 2026-11-01 20:02:18.377693839 +0000 UTC m=+33.422831030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret") pod "global-pull-secret-syncer-nn89g" (UID: "2a563d52-ad66-4e70-98e9-10266df43ddd") : object "kube-system"/"original-pull-secret" not registered
Nov 01 20:02:02.478195 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.478167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:02.478372 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.478342    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 20:02:02.478372 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.478358    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 20:02:02.478372 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.478371    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wmzf5 for pod openshift-network-diagnostics/network-check-target-q4r47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:02:02.478541 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.478416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5 podName:2782b191-652e-4551-b380-2996d67e13b5 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:18.478405469 +0000 UTC m=+33.523542673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wmzf5" (UniqueName: "kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5") pod "network-check-target-q4r47" (UID: "2782b191-652e-4551-b380-2996d67e13b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 20:02:02.550484 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.550452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:02.550484 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.550470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:02.550694 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.550512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:02.550694 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.550624    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:02.550830 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.550807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:02.550932 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:02.550912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:02.685666 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.685584    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="b1cffe778c96aa2d514ee743eae33c57c34039c59236ad3fc778e5550e31fdd2" exitCode=0
Nov 01 20:02:02.685666 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:02.685641    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"b1cffe778c96aa2d514ee743eae33c57c34039c59236ad3fc778e5550e31fdd2"}
Nov 01 20:02:03.691205 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691028    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"e5a4d87c8ae8cc462daa1adb511a74e55b204c0e1388863740f1425fc7348a33"}
Nov 01 20:02:03.691690 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691240    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"2650ef5654bf976b4cd86a32f69d130e96cf2cfd53d251b947d0d064bc824a3b"}
Nov 01 20:02:03.691690 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691272    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"597265814affcc55aa3879072c93ce248efc67e6893b7d601faa9ff51f1e3df0"}
Nov 01 20:02:03.691690 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691298    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"0fe54efeb3cc648e8a636a10ecb801998b5f724b622fe422aaa15f25cb661de8"}
Nov 01 20:02:03.691690 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691360    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"28097e69ca1ba5bfe0a2fcc67be8e12357c5d6db92f68880e3c312198c1cd87e"}
Nov 01 20:02:03.691690 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:03.691386    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"f3284f61c243ad352c5ba281e5507a2ffd358ad7b4e9469e554bdd5d9d9f6619"}
Nov 01 20:02:04.550834 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:04.550796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:04.551022 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:04.550854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:04.551022 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:04.550957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:04.551022 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:04.550976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:04.551161 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:04.551049    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:04.551161 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:04.551117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:05.698632 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:05.698598    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"6bd090add33f7133f1c25df0eaefb50f8c666f7e2ea4a66eeda1d3b458f6dc1f"}
Nov 01 20:02:05.875438 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:05.875369    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:05.875589 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:05.875490    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:02:05.875955 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:05.875939    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:06.550858 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:06.550824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:06.550858 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:06.550842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:06.551065 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:06.550824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:06.551065 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:06.550938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:06.551065 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:06.550992    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:06.551153 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:06.551083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:06.705061 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:06.705028    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="8a970286497806e9f365bbf40e1963ee7838da07205e6ff7a5ed33c7cd352384" exitCode=0
Nov 01 20:02:06.705061 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:06.705057    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"8a970286497806e9f365bbf40e1963ee7838da07205e6ff7a5ed33c7cd352384"}
Nov 01 20:02:07.710306 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.709959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" event={"ID":"73c09b58-e615-4304-b62a-a9733d309f60","Type":"ContainerStarted","Data":"c9433ee4ca822f5aff22d77c803b403497ed14bfb2e718665186c1919d51b7a5"}
Nov 01 20:02:07.710306 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.710273    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:07.711020 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.710319    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:07.713063 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.713029    2577 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="b03f48ff465fd07c0465300e85787f57e5937073ea8e0a6f35fce10d0cb64ae2" exitCode=0
Nov 01 20:02:07.713175 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.713073    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerDied","Data":"b03f48ff465fd07c0465300e85787f57e5937073ea8e0a6f35fce10d0cb64ae2"}
Nov 01 20:02:07.726327 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.726305    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:07.987969 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:07.987912    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k" podStartSLOduration=7.125263737 podStartE2EDuration="21.987898586s" totalImagesPullingTime="14.862634849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:07.987601812 +0000 UTC m=+23.032739024" watchObservedRunningTime="2026-11-01 20:02:07.987898586 +0000 UTC m=+23.033035797"
Nov 01 20:02:08.550380 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.550350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:08.550380 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.550378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:08.550681 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:08.550467    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:08.550681 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:08.550561    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:08.550681 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.550625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:08.550836 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:08.550709    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:08.718597 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.718564    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" event={"ID":"7adb9ea9-0207-415d-91c4-d85a59b4746c","Type":"ContainerStarted","Data":"38c5d994f4bb4e9388807813e62f16bbd9a1aa4c5517ead70599ced721774154"}
Nov 01 20:02:08.719042 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.718742    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:08.732782 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.732759    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:08.837388 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:08.837296    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wrz6h" podStartSLOduration=9.245181089999999 podStartE2EDuration="22.837282343s" totalImagesPullingTime="13.592101253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:08.836401553 +0000 UTC m=+23.881538767" watchObservedRunningTime="2026-11-01 20:02:08.837282343 +0000 UTC m=+23.882419555"
Nov 01 20:02:10.098608 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.098581    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nn89g"]
Nov 01 20:02:10.099054 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.098677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:10.099054 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:10.098757    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:10.101980 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.101953    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4r47"]
Nov 01 20:02:10.102099 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.102039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:10.102149 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:10.102122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:10.102605 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.102586    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqhjj"]
Nov 01 20:02:10.102688 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:10.102678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:10.102791 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:10.102774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:11.553592 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:11.553571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:11.554060 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:11.553571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:11.554060 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:11.553665    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:11.554060 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:11.553737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:11.554060 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:11.553787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:11.554060 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:11.553841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:13.550708 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:13.550670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:13.551169 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:13.550672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:13.551169 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:13.550797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:13.551169 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:13.550807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:13.551169 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:13.550866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:13.551169 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:13.550913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:15.551638 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.551387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:15.552040 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.551435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:15.552040 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:15.551736    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hqhjj" podUID="b7b66bc4-cb31-484b-b73f-80eaefeb3d78"
Nov 01 20:02:15.552040 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:15.551767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4r47" podUID="2782b191-652e-4551-b380-2996d67e13b5"
Nov 01 20:02:15.552040 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.551460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:15.552040 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:15.551827    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nn89g" podUID="2a563d52-ad66-4e70-98e9-10266df43ddd"
Nov 01 20:02:15.765486 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.765458    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeReady"
Nov 01 20:02:15.765671 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.765594    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 20:02:15.920450 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.920418    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgkq7"]
Nov 01 20:02:15.922186 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.922167    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2zq5k"]
Nov 01 20:02:15.922307 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.922282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:15.923742 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.923719    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ckjjq"]
Nov 01 20:02:15.923840 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.923828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:15.925113 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.925100    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:02:15.925217 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.925203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:15.926658 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.926641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:15.929786 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.929769    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m49w5"]
Nov 01 20:02:15.931247 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.931231    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"]
Nov 01 20:02:15.931371 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.931352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:15.932618 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.932602    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"]
Nov 01 20:02:15.932729 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.932714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:15.933969 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.933953    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"]
Nov 01 20:02:15.934065 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.934052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:15.935522 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.935484    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"]
Nov 01 20:02:15.935616 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.935598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:15.937268 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.937251    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"]
Nov 01 20:02:15.937361 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.937318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:15.938728 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.938714    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"]
Nov 01 20:02:15.938824 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.938812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:15.940331 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.940317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:15.971450 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.971426    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"]
Nov 01 20:02:15.973095 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.973078    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-646fc8f47-znjf4"]
Nov 01 20:02:15.973203 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.973187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:15.974729 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.974711    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:02:15.974827 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.974812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:15.977580 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.977549    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"]
Nov 01 20:02:15.977673 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.977625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:15.979294 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.979279    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94r67"]
Nov 01 20:02:15.979406 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.979388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:15.980587 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.980569    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 20:02:15.980752 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.980592    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 20:02:15.981112 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.981094    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8dbcd"]
Nov 01 20:02:15.981231 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.981212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:15.983073 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.983054    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2zq5k"]
Nov 01 20:02:15.983160 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.983085    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ckjjq"]
Nov 01 20:02:15.983160 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.983105    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgkq7"]
Nov 01 20:02:15.983252 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:15.983161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.031282 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031258    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cc7fv\""
Nov 01 20:02:16.031282 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031275    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 20:02:16.031422 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031282    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 20:02:16.031422 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031363    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 20:02:16.031422 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031366    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 20:02:16.031815 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.031801    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 20:02:16.032651 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.032634    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"]
Nov 01 20:02:16.033283 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.033265    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m49w5"]
Nov 01 20:02:16.033945 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.033928    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"]
Nov 01 20:02:16.034580 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.034564    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:02:16.035166 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.035151    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"]
Nov 01 20:02:16.035736 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.035721    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"]
Nov 01 20:02:16.036302 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.036286    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"]
Nov 01 20:02:16.039201 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.039183    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94r67"]
Nov 01 20:02:16.039993 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.039934    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"]
Nov 01 20:02:16.047603 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.047580    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-646fc8f47-znjf4"]
Nov 01 20:02:16.073242 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.073375 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.073375 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvbh\" (UniqueName: \"kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh\") pod \"network-check-source-6f78d59f99-xl792\" (UID: \"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:16.073375 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.073375 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2vg\" (UniqueName: \"kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.073563 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khzkp\" (UniqueName: \"kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.073563 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.073563 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdz5w\" (UniqueName: \"kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.073674 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.073674 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.073674 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.073674 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.073822 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.073822 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx77q\" (UniqueName: \"kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q\") pod \"volume-data-source-validator-58c9f5f46d-x8tnw\" (UID: \"c62b53e2-1a0a-43bb-a0b8-8181d53f2384\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:16.073916 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.073916 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.073916 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.074013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.074013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.074013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.073999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.074097 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.074097 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.074155 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.074155 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.074155 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.074260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2pc\" (UniqueName: \"kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.074260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.074260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.074346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw5r\" (UniqueName: \"kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.074346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.074346 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.074514 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.074514 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfq55\" (UniqueName: \"kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55\") pod \"kas-connection-checker-685895c7c9-2zq5k\" (UID: \"befdc8ba-b69c-4357-a5c8-fb748a4cb03e\") " pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:16.074514 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074514 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftnp\" (UniqueName: \"kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074591    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:02:16.074647 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctp6\" (UniqueName: \"kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6\") pod \"kas-connection-checker-685895c7c9-qgkq7\" (UID: \"395408fa-e36c-4283-82ee-613469a4af49\") " pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.074887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rst\" (UniqueName: \"kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.075082 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.075082 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.075082 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.074988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.075082 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72k9\" (UniqueName: \"kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.075082 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.075259 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqrc\" (UniqueName: \"kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc\") pod \"kas-connection-checker-685895c7c9-ckjjq\" (UID: \"07b921bf-2def-4eaf-8efd-a883c81f402a\") " pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:16.075259 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wmh\" (UniqueName: \"kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.075259 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.075259 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.075259 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.075215    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8dbcd"]
Nov 01 20:02:16.076127 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076109    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076215 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076201    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076265 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076250    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 20:02:16.076295 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076260    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 20:02:16.076371 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076360    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 20:02:16.076479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076456    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076457    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076653 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076503    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 20:02:16.076653 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076601    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076653 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076638    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076672    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076675    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076682    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076671    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076689    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076765 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076702    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076697    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076723    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076909    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076916    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076913    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076942    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076954    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076955    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.076978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.076948    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 20:02:16.077258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.077001    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.077258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.077027    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.077258 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.077073    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:16.081986 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.081974    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 20:02:16.126617 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.126591    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 20:02:16.126800 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.126679    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 20:02:16.126800 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.126653    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 20:02:16.127047 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127033    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 20:02:16.127320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127304    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 20:02:16.127420 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127397    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 20:02:16.127420 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127398    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7gfgj\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127426    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127402    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wmgh6\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127430    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p45lr\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127533    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127562    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ftcsj\""
Nov 01 20:02:16.127606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127597    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127617    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dmdkt\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127632    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127636    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127641    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127660    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127732    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127739    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2mhc5\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127775    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-247ns\""
Nov 01 20:02:16.127829 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127831    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mr6jl\""
Nov 01 20:02:16.128214 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127888    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4jvr\""
Nov 01 20:02:16.128214 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127926    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bwz74\""
Nov 01 20:02:16.128214 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.127981    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\""
Nov 01 20:02:16.128897 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.128881    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"]
Nov 01 20:02:16.129425 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.129411    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"]
Nov 01 20:02:16.175919 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.175898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.176013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.175922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.176013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.175938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfq55\" (UniqueName: \"kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55\") pod \"kas-connection-checker-685895c7c9-2zq5k\" (UID: \"befdc8ba-b69c-4357-a5c8-fb748a4cb03e\") " pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:16.176013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.175961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.176013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.175977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.176013 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftnp\" (UniqueName: \"kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176124    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctp6\" (UniqueName: \"kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6\") pod \"kas-connection-checker-685895c7c9-qgkq7\" (UID: \"395408fa-e36c-4283-82ee-613469a4af49\") " pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.176227 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176203    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.676183656 +0000 UTC m=+31.721320863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rst\" (UniqueName: \"kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176399    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72k9\" (UniqueName: \"kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176427    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.676411363 +0000 UTC m=+31.721548562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : configmap references non-existent config key: service-ca.crt
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176463    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.676453642 +0000 UTC m=+31.721590834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176523    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176535    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqrc\" (UniqueName: \"kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc\") pod \"kas-connection-checker-685895c7c9-ckjjq\" (UID: \"07b921bf-2def-4eaf-8efd-a883c81f402a\") " pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:16.176663 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wmh\" (UniqueName: \"kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.176571    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.676557186 +0000 UTC m=+31.721694393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvbh\" (UniqueName: \"kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh\") pod \"network-check-source-6f78d59f99-xl792\" (UID: \"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.176983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2vg\" (UniqueName: \"kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzkp\" (UniqueName: \"kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdz5w\" (UniqueName: \"kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.177320 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx77q\" (UniqueName: \"kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q\") pod \"volume-data-source-validator-58c9f5f46d-x8tnw\" (UID: \"c62b53e2-1a0a-43bb-a0b8-8181d53f2384\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2pc\" (UniqueName: \"kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.178010 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177892    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.177810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw5r\" (UniqueName: \"kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178511    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.178669 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.179105 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.178891    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:16.179105 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.178939    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.678925717 +0000 UTC m=+31.724062930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:16.179105 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.178977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.179252 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.179119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.179386 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.179349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.179486 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.179462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.179665 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.179622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.179853 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.179835    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:16.179914 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.179855    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:16.179978 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.179933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.180067 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.180053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.679996886 +0000 UTC m=+31.725134099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:16.180291 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.180271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.180467 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.180450    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:16.180617 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.180605    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.68059237 +0000 UTC m=+31.725729569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:16.180708 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.180642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.180790 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.180476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.180872 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.180555    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:16.180988 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.180976    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:16.680958806 +0000 UTC m=+31.726096003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:16.181424 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.181404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.181605 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.181451    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.181716 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.181694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.181965 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.181895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.182189 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.182162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.182839 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.182636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.182904 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.182891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.183042 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.183174 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.183246 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.183384 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.183571 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.183960 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.183935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.184650 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.184634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.234979 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.234868    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqrc\" (UniqueName: \"kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc\") pod \"kas-connection-checker-685895c7c9-ckjjq\" (UID: \"07b921bf-2def-4eaf-8efd-a883c81f402a\") " pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:16.236220 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.236077    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.236328 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.236234    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72k9\" (UniqueName: \"kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.236328 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.236284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftnp\" (UniqueName: \"kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.236682 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.236653    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfq55\" (UniqueName: \"kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55\") pod \"kas-connection-checker-685895c7c9-2zq5k\" (UID: \"befdc8ba-b69c-4357-a5c8-fb748a4cb03e\") " pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:16.236918 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.236870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctp6\" (UniqueName: \"kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6\") pod \"kas-connection-checker-685895c7c9-qgkq7\" (UID: \"395408fa-e36c-4283-82ee-613469a4af49\") " pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:16.238034 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.237990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvbh\" (UniqueName: \"kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh\") pod \"network-check-source-6f78d59f99-xl792\" (UID: \"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:16.238127 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.238091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wmh\" (UniqueName: \"kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.238801 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.238756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdz5w\" (UniqueName: \"kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.239923 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.239815    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzkp\" (UniqueName: \"kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.239923 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.239817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:16.240438 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2pc\" (UniqueName: \"kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.240438 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx77q\" (UniqueName: \"kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q\") pod \"volume-data-source-validator-58c9f5f46d-x8tnw\" (UID: \"c62b53e2-1a0a-43bb-a0b8-8181d53f2384\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:16.240604 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2vg\" (UniqueName: \"kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.240604 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw5r\" (UniqueName: \"kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.240604 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.240981 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.240965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rst\" (UniqueName: \"kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.241190 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.241170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.242561 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.242540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.245673 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.245658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:16.257359 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.257304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:16.275817 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.275755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:16.291633 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.291600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:16.297743 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.297563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:16.302951 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.302772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:16.323836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.320585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:16.335510 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.332927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:16.535419 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.535342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:16.633487 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.633464    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ckjjq"]
Nov 01 20:02:16.636544 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.636519    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2zq5k"]
Nov 01 20:02:16.636664 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.636599    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b921bf_2def_4eaf_8efd_a883c81f402a.slice/crio-2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b WatchSource:0}: Error finding container 2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b: Status 404 returned error can't find the container with id 2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b
Nov 01 20:02:16.644832 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.644810    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m49w5"]
Nov 01 20:02:16.650549 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.650522    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1946120d_acc5_423a_b8e8_eec897eb0bee.slice/crio-01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d WatchSource:0}: Error finding container 01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d: Status 404 returned error can't find the container with id 01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d
Nov 01 20:02:16.682095 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682022    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"]
Nov 01 20:02:16.682700 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:16.682793 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.682793 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:16.682887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:16.682887 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.682830    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:16.682887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:16.683025 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.682903    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.682882234 +0000 UTC m=+32.728019441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:16.683025 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.682956    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:16.683025 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.682987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:16.683025 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683006    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.682989938 +0000 UTC m=+32.728127145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.683038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683051    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683063    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.683067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.683084587 +0000 UTC m=+32.728221809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683158    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683189    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.683179509 +0000 UTC m=+32.728316723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683206    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:16.683215 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683216    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683158    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.68323845 +0000 UTC m=+32.728375642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.683256528 +0000 UTC m=+32.728393724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : configmap references non-existent config key: service-ca.crt
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.683271409 +0000 UTC m=+32.728408607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683306    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:16.683612 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:16.683332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:17.683322882 +0000 UTC m=+32.728460096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:16.688193 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.688165    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659b88a7_6607_43cf_a880_c3ee709c3862.slice/crio-08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e WatchSource:0}: Error finding container 08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e: Status 404 returned error can't find the container with id 08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e
Nov 01 20:02:16.695683 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.695660    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"]
Nov 01 20:02:16.698143 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.698117    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e840a4_f321_43d0_9864_5321d68fd28e.slice/crio-3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125 WatchSource:0}: Error finding container 3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125: Status 404 returned error can't find the container with id 3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125
Nov 01 20:02:16.707309 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.707283    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"]
Nov 01 20:02:16.712025 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.711991    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62b53e2_1a0a_43bb_a0b8_8181d53f2384.slice/crio-daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c WatchSource:0}: Error finding container daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c: Status 404 returned error can't find the container with id daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c
Nov 01 20:02:16.745338 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.745275    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"]
Nov 01 20:02:16.747611 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.747586    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce41e940_ffc1_4f08_93c8_3abd565d89e2.slice/crio-42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9 WatchSource:0}: Error finding container 42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9: Status 404 returned error can't find the container with id 42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9
Nov 01 20:02:16.749161 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.749133    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq" event={"ID":"659b88a7-6607-43cf-a880-c3ee709c3862","Type":"ContainerStarted","Data":"08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e"}
Nov 01 20:02:16.750532 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.750430    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m49w5" event={"ID":"1946120d-acc5-423a-b8e8-eec897eb0bee","Type":"ContainerStarted","Data":"01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d"}
Nov 01 20:02:16.751640 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.751617    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw" event={"ID":"c62b53e2-1a0a-43bb-a0b8-8181d53f2384","Type":"ContainerStarted","Data":"daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c"}
Nov 01 20:02:16.752782 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.752761    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ckjjq" event={"ID":"07b921bf-2def-4eaf-8efd-a883c81f402a","Type":"ContainerStarted","Data":"2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b"}
Nov 01 20:02:16.753710 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.753693    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w" event={"ID":"33e840a4-f321-43d0-9864-5321d68fd28e","Type":"ContainerStarted","Data":"3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125"}
Nov 01 20:02:16.755648 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.755625    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2zq5k" event={"ID":"befdc8ba-b69c-4357-a5c8-fb748a4cb03e","Type":"ContainerStarted","Data":"983468a794f893fffbc368cd832f2c4507c68f7fab794d7fb60a821b165df541"}
Nov 01 20:02:16.757017 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.756997    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"]
Nov 01 20:02:16.760383 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.760343    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8dbcd"]
Nov 01 20:02:16.760732 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.760708    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f7ffbf_182f_4aef_a745_fd7cd7e61c1c.slice/crio-c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18 WatchSource:0}: Error finding container c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18: Status 404 returned error can't find the container with id c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18
Nov 01 20:02:16.763077 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.763061    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a270bb_d842_48e6_a847_ceced3cc014a.slice/crio-f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f WatchSource:0}: Error finding container f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f: Status 404 returned error can't find the container with id f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f
Nov 01 20:02:16.898290 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:16.898264    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qgkq7"]
Nov 01 20:02:16.899898 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:16.899869    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395408fa_e36c_4283_82ee_613469a4af49.slice/crio-b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486 WatchSource:0}: Error finding container b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486: Status 404 returned error can't find the container with id b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486
Nov 01 20:02:17.550260 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.550234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:17.550408 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.550240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:17.550408 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.550364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:17.603556 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.603439    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 20:02:17.603556 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.603537    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\""
Nov 01 20:02:17.603807 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.603574    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\""
Nov 01 20:02:17.603807 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.603616    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 20:02:17.603807 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.603654    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\""
Nov 01 20:02:17.692353 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692558    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.692524666 +0000 UTC m=+34.737661862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : configmap references non-existent config key: service-ca.crt
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692623    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692632    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692667    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.692655273 +0000 UTC m=+34.737792478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:17.692790 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692785    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.692828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692895    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692906    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692949    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.692824075 +0000 UTC m=+34.737961282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692970    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.692981    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693040    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.693026999 +0000 UTC m=+34.738164199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.693052169 +0000 UTC m=+34.738189367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:17.693186 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693078    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.693069607 +0000 UTC m=+34.738206798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.693215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.693291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693586    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693607    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693640    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.693626251 +0000 UTC m=+34.738763457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:17.693696 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:17.693657    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:19.693648179 +0000 UTC m=+34.738785386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:17.764928 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.764889    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ckjjq" event={"ID":"07b921bf-2def-4eaf-8efd-a883c81f402a","Type":"ContainerStarted","Data":"cb6196ea4b70c0dfe4be852f2ea1bffe5900de98911e0e1cc3f9809c4c23c350"}
Nov 01 20:02:17.768225 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.767742    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qgkq7" event={"ID":"395408fa-e36c-4283-82ee-613469a4af49","Type":"ContainerStarted","Data":"b23c1ba316cd23d5bcb4c69dd868377808d41275e5b37f483342c298dd02a201"}
Nov 01 20:02:17.768225 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.767788    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qgkq7" event={"ID":"395408fa-e36c-4283-82ee-613469a4af49","Type":"ContainerStarted","Data":"b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486"}
Nov 01 20:02:17.769460 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.769437    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792" event={"ID":"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c","Type":"ContainerStarted","Data":"c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18"}
Nov 01 20:02:17.776521 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.775980    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" event={"ID":"ce41e940-ffc1-4f08-93c8-3abd565d89e2","Type":"ContainerStarted","Data":"42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9"}
Nov 01 20:02:17.781026 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.778379    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2zq5k" event={"ID":"befdc8ba-b69c-4357-a5c8-fb748a4cb03e","Type":"ContainerStarted","Data":"1f7ce6783b817111aa29c01bf8e97293e6d642614f5b82211daa4f73b683878b"}
Nov 01 20:02:17.783930 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:17.783902    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8dbcd" event={"ID":"c8a270bb-d842-48e6-a847-ceced3cc014a","Type":"ContainerStarted","Data":"f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f"}
Nov 01 20:02:18.142566 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.142328    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2zq5k" podStartSLOduration=447.142264409 podStartE2EDuration="7m27.142264409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:18.027049434 +0000 UTC m=+33.072186647" watchObservedRunningTime="2026-11-01 20:02:18.142264409 +0000 UTC m=+33.187401622"
Nov 01 20:02:18.144897 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.144682    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ckjjq" podStartSLOduration=447.144671165 podStartE2EDuration="7m27.144671165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:18.143287738 +0000 UTC m=+33.188424952" watchObservedRunningTime="2026-11-01 20:02:18.144671165 +0000 UTC m=+33.189808377"
Nov 01 20:02:18.260724 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.260297    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qgkq7" podStartSLOduration=447.260279135 podStartE2EDuration="7m27.260279135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:18.258350137 +0000 UTC m=+33.303487375" watchObservedRunningTime="2026-11-01 20:02:18.260279135 +0000 UTC m=+33.305416344"
Nov 01 20:02:18.401125 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.400977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:18.401321 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.401173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:18.402022 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:18.401974    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:02:18.402174 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:18.402048    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:50.40202932 +0000 UTC m=+65.447166525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : secret "metrics-daemon-secret" not found
Nov 01 20:02:18.421066 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.420960    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:18.473416 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.473116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:18.501942 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.501912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:18.505469 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.505448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:18.761910 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.761879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:18.891507 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:18.890670    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nn89g"]
Nov 01 20:02:18.894750 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:18.894715    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a563d52_ad66_4e70_98e9_10266df43ddd.slice/crio-ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3 WatchSource:0}: Error finding container ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3: Status 404 returned error can't find the container with id ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3
Nov 01 20:02:19.152728 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.152605    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4r47"]
Nov 01 20:02:19.159054 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:19.159006    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2782b191_652e_4551_b380_2996d67e13b5.slice/crio-2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec WatchSource:0}: Error finding container 2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec: Status 404 returned error can't find the container with id 2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec
Nov 01 20:02:19.714733 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:19.714929 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:19.715276 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.714930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:19.715276 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715122    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:19.715276 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715171418 +0000 UTC m=+38.760308627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715612    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715626    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.71561066 +0000 UTC m=+38.760747856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : configmap references non-existent config key: service-ca.crt
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715659    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715647033 +0000 UTC m=+38.760784238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715688    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715718    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715729    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715771    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715781    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715800    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715719    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715709454 +0000 UTC m=+38.760846663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715837    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715826257 +0000 UTC m=+38.760963463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715850    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715842473 +0000 UTC m=+38.760979669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715855    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715865    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715855865 +0000 UTC m=+38.760993056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:19.716009 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:19.715882    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.715874331 +0000 UTC m=+38.761011527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:19.803668 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.803132    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw" event={"ID":"c62b53e2-1a0a-43bb-a0b8-8181d53f2384","Type":"ContainerStarted","Data":"a3d16d32a9a465787bc7d6ae897a045001dd352a0264ef8011b4aa331a314a2f"}
Nov 01 20:02:19.805438 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.805398    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nn89g" event={"ID":"2a563d52-ad66-4e70-98e9-10266df43ddd","Type":"ContainerStarted","Data":"ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3"}
Nov 01 20:02:19.807585 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.807562    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4r47" event={"ID":"2782b191-652e-4551-b380-2996d67e13b5","Type":"ContainerStarted","Data":"2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec"}
Nov 01 20:02:19.919836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:19.919722    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw" podStartSLOduration=397.443111449 podStartE2EDuration="6m39.919701817s" totalImagesPullingTime="2.476590368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:19.918672184 +0000 UTC m=+34.963809433" watchObservedRunningTime="2026-11-01 20:02:19.919701817 +0000 UTC m=+34.964839023"
Nov 01 20:02:21.822653 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:21.821893    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m49w5" event={"ID":"1946120d-acc5-423a-b8e8-eec897eb0bee","Type":"ContainerStarted","Data":"c397e07f0bf754b77cf9bb6dd2523a383e3310641c278c1161b79a2b0ee846c3"}
Nov 01 20:02:21.826699 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:21.825820    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/0.log"
Nov 01 20:02:21.826699 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:21.825861    2577 generic.go:356] "Generic (PLEG): container finished" podID="ce41e940-ffc1-4f08-93c8-3abd565d89e2" containerID="ed879671c00e630a0777a41c4f27ccef8ac066f6a1d27a0f653bb36b1bb2d541" exitCode=255
Nov 01 20:02:21.826699 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:21.825936    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" event={"ID":"ce41e940-ffc1-4f08-93c8-3abd565d89e2","Type":"ContainerDied","Data":"ed879671c00e630a0777a41c4f27ccef8ac066f6a1d27a0f653bb36b1bb2d541"}
Nov 01 20:02:21.826699 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:21.826356    2577 scope.go:122] "RemoveContainer" containerID="ed879671c00e630a0777a41c4f27ccef8ac066f6a1d27a0f653bb36b1bb2d541"
Nov 01 20:02:22.055698 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.055642    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-m49w5" podStartSLOduration=401.47750149 podStartE2EDuration="6m46.055626684s" totalImagesPullingTime="4.578125194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:21.939314948 +0000 UTC m=+36.984452162" watchObservedRunningTime="2026-11-01 20:02:22.055626684 +0000 UTC m=+37.100763896"
Nov 01 20:02:22.835595 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.835525    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8dbcd" event={"ID":"c8a270bb-d842-48e6-a847-ceced3cc014a","Type":"ContainerStarted","Data":"6f54dab916ce02c23cf60cafdbb222cc143f48214dc8e279e9ef31ec1bec31ea"}
Nov 01 20:02:22.839846 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.838454    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sr5gb"]
Nov 01 20:02:22.841396 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.840801    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq" event={"ID":"659b88a7-6607-43cf-a880-c3ee709c3862","Type":"ContainerStarted","Data":"4738e02a9c8bb2ee92737319a41249ee5f8f7d8388505be1b5524188fde211d5"}
Nov 01 20:02:22.841396 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.840925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.844790 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.844767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:22.845364 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.845334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/0.log"
Nov 01 20:02:22.845443 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.845373    2577 generic.go:356] "Generic (PLEG): container finished" podID="ce41e940-ffc1-4f08-93c8-3abd565d89e2" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14" exitCode=255
Nov 01 20:02:22.845479 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.845461    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" event={"ID":"ce41e940-ffc1-4f08-93c8-3abd565d89e2","Type":"ContainerDied","Data":"e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"}
Nov 01 20:02:22.845549 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.845518    2577 scope.go:122] "RemoveContainer" containerID="ed879671c00e630a0777a41c4f27ccef8ac066f6a1d27a0f653bb36b1bb2d541"
Nov 01 20:02:22.845887 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.845873    2577 scope.go:122] "RemoveContainer" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"
Nov 01 20:02:22.846081 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:22.846058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:22.851922 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.851167    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w" event={"ID":"33e840a4-f321-43d0-9864-5321d68fd28e","Type":"ContainerStarted","Data":"2fbcf5e0b5c75dd51478eaf580ced053e27dfb06c37aa5f728d299facf3715b6"}
Nov 01 20:02:22.894299 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.894255    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 20:02:22.894477 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.894387    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\""
Nov 01 20:02:22.940270 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.940235    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 20:02:22.942255 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.942227    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sr5gb"]
Nov 01 20:02:22.953381 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.951734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.953381 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.952024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.953381 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.952068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.953381 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.952719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.953381 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.952762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bc6\" (UniqueName: \"kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:22.958956 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:22.958862    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8dbcd" podStartSLOduration=2.374637481 podStartE2EDuration="7.95884165s" totalImagesPullingTime="5.584204169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:22.951095415 +0000 UTC m=+37.996232651" watchObservedRunningTime="2026-11-01 20:02:22.95884165 +0000 UTC m=+38.003978864"
Nov 01 20:02:23.053854 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.053816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054023 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.053958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054023 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.053996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054133 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.054017    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054133 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.054035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054133 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.054077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bc6\" (UniqueName: \"kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.054276 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.054133    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:23.054276 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.054194    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.554176704 +0000 UTC m=+38.599313899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:23.054384 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.054343    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:23.054433 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.054393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:23.554382067 +0000 UTC m=+38.599519259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:23.054684 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.054668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.096233 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.096158    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:02:23.099327 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.099302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.129405 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.129372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bc6\" (UniqueName: \"kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.160035 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.160002    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:02:23.185939 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.185884    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq" podStartSLOduration=399.987603506 podStartE2EDuration="6m45.18587176s" totalImagesPullingTime="5.198268254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:23.185296701 +0000 UTC m=+38.230433934" watchObservedRunningTime="2026-11-01 20:02:23.18587176 +0000 UTC m=+38.231008973"
Nov 01 20:02:23.259454 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.259417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.259645 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.259472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.259645 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.259592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.259645 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.259608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.303150 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.302866    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w" podStartSLOduration=395.942760245 podStartE2EDuration="6m41.302845836s" totalImagesPullingTime="5.360085591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:23.300923426 +0000 UTC m=+38.346060664" watchObservedRunningTime="2026-11-01 20:02:23.302845836 +0000 UTC m=+38.347983050"
Nov 01 20:02:23.360952 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.360875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.360952 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.360922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.361128 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.361017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.361128 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.361062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.361885 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.361855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.362065 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.362034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.362121 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.362080    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.421606 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.420342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.570296 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.570253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.570458 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.570323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:23.570458 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.570359    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:23.570458 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.570432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:24.570413266 +0000 UTC m=+39.615550471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:23.570633 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.570460    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:23.570633 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.570551    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:24.570533802 +0000 UTC m=+39.615671033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:23.612801 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.611436    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"]
Nov 01 20:02:23.613836 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.613812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:23.619007 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.617880    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"]
Nov 01 20:02:23.620077 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.620058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:23.688830 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.688804    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 20:02:23.689007 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.688943    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:23.689081 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.689050    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f22l2\""
Nov 01 20:02:23.690469 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.690440    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"]
Nov 01 20:02:23.713800 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.713772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:23.716126 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.716101    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"]
Nov 01 20:02:23.772763 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:23.772859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:23.772859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:23.772859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsx89\" (UniqueName: \"kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89\") pod \"migrator-66df5d496b-c7hsh\" (UID: \"35dd1a4d-103c-4343-a834-3479d75e59e0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:23.772859 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259bj\" (UniqueName: \"kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj\") pod \"migrator-66df5d496b-996rs\" (UID: \"3014c67d-90f7-41d9-8a27-1657568714cd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:23.772983 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.772887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:23.772983 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.772903    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:23.772983 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.772958    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:23.772983 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.772974    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.772954593 +0000 UTC m=+46.818091801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:23.773131 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773012    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:23.773131 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773045    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773034607 +0000 UTC m=+46.818171813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:23.773131 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773075    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:23.773131 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773098    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:23.773131 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773085    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773068182 +0000 UTC m=+46.818205392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773137    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773126353 +0000 UTC m=+46.818263560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.773184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.773229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773270    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773261643 +0000 UTC m=+46.818398838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : configmap references non-existent config key: service-ca.crt
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.773271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.773300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:23.773321 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773321    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773324    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773378    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773396    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773360    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773348746 +0000 UTC m=+46.818485962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773437544 +0000 UTC m=+46.818574749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:23.773653 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.773509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.773484463 +0000 UTC m=+46.818621666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:23.856508 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.856467    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792" event={"ID":"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c","Type":"ContainerStarted","Data":"93a4bd7bb974841238d88adfe3ebef5277cf1b642138423d9703af941302cce8"}
Nov 01 20:02:23.860131 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.860103    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:23.860823 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.860804    2577 scope.go:122] "RemoveContainer" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"
Nov 01 20:02:23.860993 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:23.860972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:23.874967 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.874895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx89\" (UniqueName: \"kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89\") pod \"migrator-66df5d496b-c7hsh\" (UID: \"35dd1a4d-103c-4343-a834-3479d75e59e0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:23.874967 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.874934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259bj\" (UniqueName: \"kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj\") pod \"migrator-66df5d496b-996rs\" (UID: \"3014c67d-90f7-41d9-8a27-1657568714cd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:23.933044 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.932949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx89\" (UniqueName: \"kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89\") pod \"migrator-66df5d496b-c7hsh\" (UID: \"35dd1a4d-103c-4343-a834-3479d75e59e0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:23.933044 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.932949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259bj\" (UniqueName: \"kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj\") pod \"migrator-66df5d496b-996rs\" (UID: \"3014c67d-90f7-41d9-8a27-1657568714cd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:23.940546 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.935793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:23.977119 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:23.977042    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792" podStartSLOduration=398.35845725 podStartE2EDuration="6m44.977022523s" totalImagesPullingTime="6.618565273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:23.975018373 +0000 UTC m=+39.020155586" watchObservedRunningTime="2026-11-01 20:02:23.977022523 +0000 UTC m=+39.022159737"
Nov 01 20:02:24.096678 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.096640    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:02:24.100708 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:24.100677    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34b1735_2c5a_442b_a067_2c165eb00003.slice/crio-4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7 WatchSource:0}: Error finding container 4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7: Status 404 returned error can't find the container with id 4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7
Nov 01 20:02:24.228333 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.228303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:24.317391 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.317358    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"]
Nov 01 20:02:24.319637 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:24.319602    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dd1a4d_103c_4343_a834_3479d75e59e0.slice/crio-28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec WatchSource:0}: Error finding container 28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec: Status 404 returned error can't find the container with id 28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec
Nov 01 20:02:24.582654 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.582617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:24.582843 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.582675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:24.582843 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:24.582750    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:24.582843 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:24.582804    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:24.582843 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:24.582824    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:26.582804062 +0000 UTC m=+41.627941267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:24.584674 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:24.582853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:26.582838943 +0000 UTC m=+41.627976147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:24.605393 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.605313    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"]
Nov 01 20:02:24.614339 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:24.614309    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3014c67d_90f7_41d9_8a27_1657568714cd.slice/crio-9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6 WatchSource:0}: Error finding container 9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6: Status 404 returned error can't find the container with id 9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6
Nov 01 20:02:24.868261 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.868168    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs" event={"ID":"3014c67d-90f7-41d9-8a27-1657568714cd","Type":"ContainerStarted","Data":"9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6"}
Nov 01 20:02:24.869437 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.869400    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh" event={"ID":"35dd1a4d-103c-4343-a834-3479d75e59e0","Type":"ContainerStarted","Data":"28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec"}
Nov 01 20:02:24.871224 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.871200    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" event={"ID":"c34b1735-2c5a-442b-a067-2c165eb00003","Type":"ContainerStarted","Data":"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"}
Nov 01 20:02:24.871349 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.871248    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" event={"ID":"c34b1735-2c5a-442b-a067-2c165eb00003","Type":"ContainerStarted","Data":"4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7"}
Nov 01 20:02:24.872823 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.872794    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4r47" event={"ID":"2782b191-652e-4551-b380-2996d67e13b5","Type":"ContainerStarted","Data":"74cc21f46faa39b0b8ca220a1c6456ff87e007bf27498bb417bab52eae81c2f5"}
Nov 01 20:02:24.988066 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:24.988005    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" podStartSLOduration=1.98798773 podStartE2EDuration="1.98798773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:24.986881058 +0000 UTC m=+40.032018271" watchObservedRunningTime="2026-11-01 20:02:24.98798773 +0000 UTC m=+40.033124945"
Nov 01 20:02:25.101060 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.101001    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q4r47" podStartSLOduration=33.604022859 podStartE2EDuration="39.100982762s" totalImagesPullingTime="5.496959903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:25.10008921 +0000 UTC m=+40.145226425" watchObservedRunningTime="2026-11-01 20:02:25.100982762 +0000 UTC m=+40.146119974"
Nov 01 20:02:25.207014 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.206937    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tjssh"]
Nov 01 20:02:25.209143 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.209120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.265569 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.265537    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 20:02:25.265569 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.265546    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 20:02:25.266199 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.265546    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fqldw\""
Nov 01 20:02:25.266199 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.265621    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 20:02:25.266199 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.265627    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 20:02:25.267266 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.267243    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tjssh"]
Nov 01 20:02:25.389018 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.388975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.389018 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.389022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjf4\" (UniqueName: \"kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.389256 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.389225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.489871 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.489836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.489871 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.489878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjf4\" (UniqueName: \"kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.490104 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.489986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.490703 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.490679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.492740 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.492713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.549893 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.549861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjf4\" (UniqueName: \"kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.822221 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.822111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:25.879192 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:25.879165    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:26.206192 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:26.206102    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tjssh"]
Nov 01 20:02:26.210582 ip-10-0-142-223 kubenswrapper[2577]: W1101 20:02:26.210555    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f29811_63a6_4c22_8b23_ad93c7277b3b.slice/crio-1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37 WatchSource:0}: Error finding container 1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37: Status 404 returned error can't find the container with id 1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37
Nov 01 20:02:26.303030 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:26.303007    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:26.303158 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:26.303059    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:26.304016 ip-10-0-142-223 kubenswrapper[2577]: I1101 20:02:26.303984    2577 scope.go:122] "RemoveContainer" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"
Nov 01 20:02:26.304327 ip-10-0-142-223 kubenswrapper[2577]: E1101 20:02:26.304300    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:26.557828 ip-10-0-142-223 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 20:02:26.570887 ip-10-0-142-223 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 20:02:26.571053 ip-10-0-142-223 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 20:02:26.571224 ip-10-0-142-223 systemd[1]: kubelet.service: Consumed 3.078s CPU time.
Nov 01 20:02:26.584017 ip-10-0-142-223 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 20:02:26.646618 ip-10-0-142-223 kubenswrapper[5066]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:02:26.646618 ip-10-0-142-223 kubenswrapper[5066]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 20:02:26.646618 ip-10-0-142-223 kubenswrapper[5066]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:02:26.646618 ip-10-0-142-223 kubenswrapper[5066]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 20:02:26.648625 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648544    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.648625 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648624    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.648625 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648628    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648631    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648634    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648637    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648640    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648642    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648645    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648648    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648650    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648653    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648656    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648658    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648661    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648664    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648667    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648673    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648676    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648679    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648682    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.648717 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648684    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648687    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648690    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648692    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648695    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648698    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648701    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648703    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648706    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648708    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648711    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648714    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648716    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648719    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648722    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648724    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648727    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648729    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648732    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648735    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.649175 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648737    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648740    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648743    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648745    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648748    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648751    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648754    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648757    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648759    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648762    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648766    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648768    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648771    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648773    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648777    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648780    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648783    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648785    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648788    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.649680 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648791    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648794    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648797    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648799    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648802    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648804    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648807    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648809    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648812    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648815    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648817    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648820    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648823    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648825    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648828    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648830    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648832    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648835    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648838    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648840    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.650139 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648843    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648846    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648848    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648851    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648853    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648856    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648859    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648862    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648866    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648870    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648873    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648875    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648878    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648881    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648883    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648886    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648888    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648890    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648894    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.650640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648898    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648900    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648903    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648905    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648908    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648910    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.648913    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649676    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649684    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649687    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649691    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649694    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649696    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649699    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649702    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649705    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649708    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649712    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649716    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.651086 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649719    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649722    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649725    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649728    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649730    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649733    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649736    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649738    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649741    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649743    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649746    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649748    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649751    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649753    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649756    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649759    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649761    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649764    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649767    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649770    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.651582 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649772    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649775    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649778    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649780    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649783    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649785    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649788    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649790    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649793    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649796    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649798    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649801    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649803    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649806    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649809    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649811    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649814    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649817    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649819    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.652062 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649821    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649825    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649828    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649832    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649834    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649838    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649840    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649843    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649846    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649849    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649851    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649854    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649857    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649859    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649862    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649864    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649867    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649869    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649872    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649875    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.652559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649877    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649880    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649883    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649885    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649888    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649890    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649894    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649897    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649899    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649902    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649907    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649910    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649912    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649915    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649917    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649920    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649922    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649925    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649928    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.653072 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649930    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649933    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649936    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649938    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649941    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649943    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649945    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649948    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649951    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649953    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649955    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649958    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649960    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649963    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649965    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.649969    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650432    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650437    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650440    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650443    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.653535 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650446    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650448    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650451    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650455    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650458    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650461    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650463    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650466    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650468    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650471    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650475    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650479    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650482    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650484    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650487    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650509    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650513    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650517    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650520    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.654045 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650523    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650526    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650529    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650531    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650534    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650537    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650539    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650542    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650544    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650548    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650550    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650553    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650556    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650558    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650561    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650563    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650567    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650570    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650573    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.654539 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650576    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650580    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650583    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650585    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650588    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650591    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650593    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650596    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650598    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650601    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650603    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650606    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650609    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650611    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650614    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650616    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650619    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650621    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650623    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650626    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.654990 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650628    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650631    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650633    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650637    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650640    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650642    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650645    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650648    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650651    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650654    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650657    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650659    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650662    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650665    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650667    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650669    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650672    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650675    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650677    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650680    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.655455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650682    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650685    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650687    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650690    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650693    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650696    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650698    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650701    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650703    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650706    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650728    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650731    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650734    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650737    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650740    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650743    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650746    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650749    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650752    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650755    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.655944 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650757    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650762    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650764    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.650767    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650857    5066 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650869    5066 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650875    5066 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650880    5066 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650884    5066 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650887    5066 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650892    5066 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650899    5066 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650902    5066 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650906    5066 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650909    5066 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650913    5066 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650918    5066 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650921    5066 flags.go:64] FLAG: --cgroup-root=""
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650924    5066 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650927    5066 flags.go:64] FLAG: --client-ca-file=""
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650929    5066 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650933    5066 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650938    5066 flags.go:64] FLAG: --cluster-domain=""
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650941    5066 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 20:02:26.656412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650944    5066 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650947    5066 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650950    5066 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650954    5066 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650957    5066 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650961    5066 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650964    5066 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650967    5066 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650970    5066 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650974    5066 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650977    5066 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650980    5066 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650984    5066 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650987    5066 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650990    5066 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650993    5066 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650996    5066 flags.go:64] FLAG: --enable-server="true"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.650999    5066 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651003    5066 flags.go:64] FLAG: --event-burst="100"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651006    5066 flags.go:64] FLAG: --event-qps="50"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651010    5066 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651013    5066 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651016    5066 flags.go:64] FLAG: --eviction-hard=""
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651020    5066 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 20:02:26.656991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651024    5066 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651027    5066 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651030    5066 flags.go:64] FLAG: --eviction-soft=""
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651033    5066 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651036    5066 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651039    5066 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651042    5066 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651045    5066 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651048    5066 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651051    5066 flags.go:64] FLAG: --feature-gates=""
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651055    5066 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651058    5066 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651061    5066 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651064    5066 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651068    5066 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651071    5066 flags.go:64] FLAG: --help="false"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651074    5066 flags.go:64] FLAG: --hostname-override="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651079    5066 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651082    5066 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651085    5066 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651089    5066 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651092    5066 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651095    5066 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 20:02:26.657586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651098    5066 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651101    5066 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651104    5066 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651107    5066 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651110    5066 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651113    5066 flags.go:64] FLAG: --kube-reserved=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651116    5066 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651118    5066 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651121    5066 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651125    5066 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651128    5066 flags.go:64] FLAG: --lock-file=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651132    5066 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651134    5066 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651137    5066 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651142    5066 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651145    5066 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651149    5066 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651151    5066 flags.go:64] FLAG: --logging-format="text"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651154    5066 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651157    5066 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651160    5066 flags.go:64] FLAG: --manifest-url=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651163    5066 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651168    5066 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651170    5066 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651175    5066 flags.go:64] FLAG: --max-pods="110"
Nov 01 20:02:26.658133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651178    5066 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651181    5066 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651184    5066 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651187    5066 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651190    5066 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651192    5066 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651195    5066 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651201    5066 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651204    5066 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651207    5066 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651210    5066 flags.go:64] FLAG: --pod-cidr=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651212    5066 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651215    5066 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651218    5066 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651221    5066 flags.go:64] FLAG: --port="10250"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651224    5066 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651228    5066 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-084d411af08466a80"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651231    5066 flags.go:64] FLAG: --qos-reserved=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651234    5066 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651238    5066 flags.go:64] FLAG: --register-node="true"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651240    5066 flags.go:64] FLAG: --register-with-taints=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651244    5066 flags.go:64] FLAG: --registry-burst="10"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651247    5066 flags.go:64] FLAG: --registry-qps="5"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651250    5066 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651253    5066 flags.go:64] FLAG: --reserved-memory=""
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651256    5066 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 20:02:26.658783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651259    5066 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651262    5066 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651265    5066 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651268    5066 flags.go:64] FLAG: --runonce="false"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651271    5066 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651274    5066 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651278    5066 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651281    5066 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651284    5066 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651287    5066 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651290    5066 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651292    5066 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651295    5066 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651298    5066 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651301    5066 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651304    5066 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651307    5066 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651310    5066 flags.go:64] FLAG: --system-cgroups=""
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651312    5066 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651317    5066 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651320    5066 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651323    5066 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651329    5066 flags.go:64] FLAG: --tls-min-version=""
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651332    5066 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651335    5066 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 20:02:26.659461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651337    5066 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651340    5066 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651343    5066 flags.go:64] FLAG: --v="2"
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651347    5066 flags.go:64] FLAG: --version="false"
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651351    5066 flags.go:64] FLAG: --vmodule=""
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651355    5066 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651358    5066 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651486    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651502    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651507    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651510    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651513    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651515    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651518    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651528    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651530    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651533    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651536    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651540    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651544    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.660077 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651547    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651550    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651553    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651556    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651559    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651562    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651564    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651567    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651569    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651574    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651577    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651579    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651582    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651585    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651588    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651591    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651593    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651597    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651599    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651602    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.660620 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651604    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651607    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651611    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651614    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651617    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651620    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651622    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651626    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651629    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651631    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651634    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651636    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651639    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651643    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651646    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651648    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651651    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651653    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651656    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.661161 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651659    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651661    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651665    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651668    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651671    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651673    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651676    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651678    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651681    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651683    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651686    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651688    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651691    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651694    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651696    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651699    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651701    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651704    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651706    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651708    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.661893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651711    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651714    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651717    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651720    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651722    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651725    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651728    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651731    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651733    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651736    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651739    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651741    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651744    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651746    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651750    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651752    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651755    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651757    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651760    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651762    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.662568 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651765    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651767    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651770    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651772    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651775    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651777    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651780    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651782    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651785    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651788    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651790    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651793    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651795    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651798    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.651823    5066 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651881    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.663118 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651886    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651888    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651891    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651895    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651898    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651901    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651903    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651906    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651909    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651911    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651915    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651918    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651921    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651923    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651926    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651928    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651931    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651933    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651936    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651939    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.663660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651941    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651944    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651953    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651957    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651959    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651962    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651964    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651967    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651969    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651972    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651974    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651978    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651980    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651983    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651986    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651989    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651992    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651995    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.651997    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652000    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.664515 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652002    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652005    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652009    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652012    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652014    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652017    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652019    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652022    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652024    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652027    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652030    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652033    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652035    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652037    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652040    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652043    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652045    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652048    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652050    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.665265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652053    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652056    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652059    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652063    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652066    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652070    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652073    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652076    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652080    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652083    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652085    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652088    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652090    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652093    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652096    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652099    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652103    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652106    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652109    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.665911 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652111    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652113    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652116    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652119    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652121    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652124    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652127    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652130    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652134    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652137    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652140    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652142    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652145    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652147    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652150    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652153    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652155    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652158    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652161    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.666410 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652164    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652166    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652170    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652172    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652175    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652177    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652180    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.652182    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.655975    5066 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.655992    5066 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.656010    5066 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.656015    5066 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656082    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656088    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656091    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656094    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656097    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656101    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656106    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.666873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656110    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656114    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656117    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656120    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656123    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656126    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656129    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656131    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656134    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656137    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656139    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656142    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656145    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656148    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656150    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656153    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656155    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656158    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656161    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656163    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.667326 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656166    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656169    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656179    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656183    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656185    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656188    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656191    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656193    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656196    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656198    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656201    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656204    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656207    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656209    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656212    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656215    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656217    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656220    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656222    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.667922 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656225    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656228    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656230    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656233    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656235    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656238    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656240    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656243    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656246    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656248    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656251    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656253    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656256    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656258    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656261    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656263    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656272    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656276    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656279    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.668394 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656282    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656285    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656288    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656291    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656294    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656297    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656300    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656303    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656305    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656308    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656310    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656313    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656315    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656318    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656320    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656323    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656325    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656328    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656330    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656333    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.668873 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656336    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656338    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656340    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656343    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656345    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656348    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656350    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656352    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656355    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656357    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656365    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656368    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656370    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656373    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656375    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656377    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656380    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656383    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656385    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656388    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.669343 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656390    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.656417    5066 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656632    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656644    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656647    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656650    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656653    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656655    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656658    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656661    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656663    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656666    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656669    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656671    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656674    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656676    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.669825 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656679    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656681    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656684    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656686    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656689    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656691    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656694    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656707    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656709    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656712    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656715    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656717    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656720    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656722    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656725    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656727    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656730    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656732    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656735    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.670200 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656737    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656740    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656742    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656745    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656747    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656750    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656752    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656755    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656757    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656760    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656763    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656765    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656768    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656771    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656775    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656778    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656781    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656783    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656786    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656789    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.670660 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656791    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656799    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656802    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656805    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656807    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656810    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656812    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656814    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656817    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656819    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656822    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656824    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656827    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656830    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656832    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656834    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656837    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656839    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656842    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656844    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.671229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656847    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656849    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656852    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656854    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656860    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656862    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656865    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656867    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656870    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656873    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656875    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656878    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656880    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656883    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656891    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656894    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656896    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656898    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656901    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656903    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.671881 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656906    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656909    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656911    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656914    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656916    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656919    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656921    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656924    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656926    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656930    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656934    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656937    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656940    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.656964    5066 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.656999    5066 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657004    5066 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 20:02:26.672537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657007    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657011    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657013    5066 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657016    5066 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657018    5066 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657021    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657023    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657026    5066 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657028    5066 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657031    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657033    5066 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657041    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657044    5066 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657047    5066 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657049    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657052    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657054    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657056    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657059    5066 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 20:02:26.673265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657061    5066 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657064    5066 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657066    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657069    5066 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657071    5066 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657074    5066 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657076    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657079    5066 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657081    5066 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657084    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657086    5066 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657089    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657091    5066 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657094    5066 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657096    5066 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657099    5066 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657103    5066 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657106    5066 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657109    5066 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657112    5066 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 20:02:26.673910 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657115    5066 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657117    5066 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657120    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657123    5066 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657125    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657133    5066 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657136    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657140    5066 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657142    5066 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657145    5066 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657148    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657151    5066 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657153    5066 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657156    5066 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657158    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657161    5066 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657163    5066 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657166    5066 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657168    5066 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 20:02:26.674559 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657171    5066 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657173    5066 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657176    5066 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657178    5066 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657181    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657183    5066 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657186    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657188    5066 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657191    5066 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657193    5066 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657196    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657198    5066 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657201    5066 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657203    5066 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657206    5066 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657208    5066 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657211    5066 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657213    5066 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657216    5066 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 20:02:26.675133 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657226    5066 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657232    5066 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657236    5066 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657240    5066 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657244    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657247    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657250    5066 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657254    5066 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657260    5066 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657266    5066 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657269    5066 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657272    5066 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657275    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657277    5066 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657279    5066 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657282    5066 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657285    5066 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657287    5066 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657289    5066 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657292    5066 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 20:02:26.675727 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657294    5066 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657297    5066 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657300    5066 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657302    5066 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657305    5066 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657308    5066 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:26.657311    5066 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.657362    5066 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.658532    5066 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.658586    5066 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.658777    5066 server.go:1014] "Starting client certificate rotation"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.658917    5066 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.659020    5066 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:56:46 +0000 UTC" deadline="2028-05-18 12:05:31.947084077 +0000 UTC"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.659073    5066 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13528h3m5.288013187s"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.659306    5066 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.660955    5066 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 20:02:26.676258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.663150    5066 log.go:25] "Validated CRI v1 runtime API"
Nov 01 20:02:26.676696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.665205    5066 log.go:25] "Validated CRI v1 image API"
Nov 01 20:02:26.676696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.665945    5066 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 20:02:26.676696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.667905    5066 fs.go:135] Filesystem UUIDs: map[65b92ded-d54c-42af-b003-e9ff6d74fe72:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89243e4c-36cf-4a76-82d6-680cc6d0c62f:/dev/nvme0n1p4]
Nov 01 20:02:26.676819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.667923    5066 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/983468a794f893fffbc368cd832f2c4507c68f7fab794d7fb60a821b165df541/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/983468a794f893fffbc368cd832f2c4507c68f7fab794d7fb60a821b165df541/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/067dc6af-f795-489f-820d-b81b63250a86/volumes/kubernetes.io~projected/kube-api-access-mtwb4:{mountpoint:/var/lib/kubelet/pods/067dc6af-f795-489f-820d-b81b63250a86/volumes/kubernetes.io~projected/kube-api-access-mtwb4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07b921bf-2def-4eaf-8efd-a883c81f402a/volumes/kubernetes.io~projected/kube-api-access-hbqrc:{mountpoint:/var/lib/kubelet/pods/07b921bf-2def-4eaf-8efd-a883c81f402a/volumes/kubernetes.io~projected/kube-api-access-hbqrc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0391dc-0199-4138-97d2-4db3371dda0f/volumes/kubernetes.io~projected/kube-api-access-g6rj4:{mountpoint:/var/lib/kubelet/pods/0a0391dc-0199-4138-97d2-4db3371dda0f/volumes/kubernetes.io~projected/kube-api-access-g6rj4 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a4a52df-9a7b-4332-92ec-8ddc8d299bad/volumes/kubernetes.io~projected/kube-api-access-k8rst:{mountpoint:/var/lib/kubelet/pods/0a4a52df-9a7b-4332-92ec-8ddc8d299bad/volumes/kubernetes.io~projected/kube-api-access-k8rst major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1495ed93-b519-468e-8dd2-e40df2c3e979/volumes/kubernetes.io~projected/kube-api-access-8w2vg:{mountpoint:/var/lib/kubelet/pods/1495ed93-b519-468e-8dd2-e40df2c3e979/volumes/kubernetes.io~projected/kube-api-access-8w2vg major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~projected/kube-api-access-2s2pc:{mountpoint:/var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~projected/kube-api-access-2s2pc major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc/volumes/kubernetes.io~projected/kube-api-access-mz6bb:{mountpoint:/var/lib/kubelet/pods/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc/volumes/kubernetes.io~projected/kube-api-access-mz6bb major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~projected/kube-api-access-z72k9:{mountpoint:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~projected/kube-api-access-z72k9 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c/volumes/kubernetes.io~projected/kube-api-access-hcvbh:{mountpoint:/var/lib/kubelet/pods/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c/volumes/kubernetes.io~projected/kube-api-access-hcvbh major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2782b191-652e-4551-b380-2996d67e13b5/volumes/kubernetes.io~projected/kube-api-access-wmzf5:{mountpoint:/var/lib/kubelet/pods/2782b191-652e-4551-b380-2996d67e13b5/volumes/kubernetes.io~projected/kube-api-access-wmzf5 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a563d52-ad66-4e70-98e9-10266df43ddd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2a563d52-ad66-4e70-98e9-10266df43ddd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3014c67d-90f7-41d9-8a27-1657568714cd/volumes/kubernetes.io~projected/kube-api-access-259bj:{mountpoint:/var/lib/kubelet/pods/3014c67d-90f7-41d9-8a27-1657568714cd/volumes/kubernetes.io~projected/kube-api-access-259bj major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~projected/kube-api-access-96wmh:{mountpoint:/var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~projected/kube-api-access-96wmh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35dd1a4d-103c-4343-a834-3479d75e59e0/volumes/kubernetes.io~projected/kube-api-access-gsx89:{mountpoint:/var/lib/kubelet/pods/35dd1a4d-103c-4343-a834-3479d75e59e0/volumes/kubernetes.io~projected/kube-api-access-gsx89 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395408fa-e36c-4283-82ee-613469a4af49/volumes/kubernetes.io~projected/kube-api-access-zctp6:{mountpoint:/var/lib/kubelet/pods/395408fa-e36c-4283-82ee-613469a4af49/volumes/kubernetes.io~projected/kube-api-access-zctp6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42486c5c-cb87-4f5e-96b3-f957803d0b75/volumes/kubernetes.io~projected/kube-api-access-q8bc6:{mountpoint:/var/lib/kubelet/pods/42486c5c-cb87-4f5e-96b3-f957803d0b75/volumes/kubernetes.io~projected/kube-api-access-q8bc6 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/kube-api-access-dhc4c:{mountpoint:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/kube-api-access-dhc4c major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~projected/kube-api-access-hftnp:{mountpoint:/var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~projected/kube-api-access-hftnp major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~projected/kube-api-access-68tcn:{mountpoint:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~projected/kube-api-access-68tcn major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adb9ea9-0207-415d-91c4-d85a59b4746c/volumes/kubernetes.io~projected/kube-api-access-tsg4k:{mountpoint:/var/lib/kubelet/pods/7adb9ea9-0207-415d-91c4-d85a59b4746c/volumes/kubernetes.io~projected/kube-api-access-tsg4k major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/kube-api-access-59pbf:{mountpoint:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/kube-api-access-59pbf major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~projected/kube-api-access-vrgx5:{mountpoint:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~projected/kube-api-access-vrgx5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ff9c3f-675e-4693-adb0-9bdf4308347c/volumes/kubernetes.io~projected/kube-api-access-v42mm:{mountpoint:/var/lib/kubelet/pods/b5ff9c3f-675e-4693-adb0-9bdf4308347c/volumes/kubernetes.io~projected/kube-api-access-v42mm major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7b66bc4-cb31-484b-b73f-80eaefeb3d78/volumes/kubernetes.io~projected/kube-api-access-wf9nv:{mountpoint:/var/lib/kubelet/pods/b7b66bc4-cb31-484b-b73f-80eaefeb3d78/volumes/kubernetes.io~projected/kube-api-access-wf9nv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befdc8ba-b69c-4357-a5c8-fb748a4cb03e/volumes/kubernetes.io~projected/kube-api-access-nfq55:{mountpoint:/var/lib/kubelet/pods/befdc8ba-b69c-4357-a5c8-fb748a4cb03e/volumes/kubernetes.io~projected/kube-api-access-nfq55 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c34b1735-2c5a-442b-a067-2c165eb00003/volumes/kubernetes.io~projected/kube-api-access-wnn56:{mountpoint:/var/lib/kubelet/pods/c34b1735-2c5a-442b-a067-2c165eb00003/volumes/kubernetes.io~projected/kube-api-access-wnn56 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad/volumes/kubernetes.io~projected/kube-api-access-qhw5r:{mountpoint:/var/lib/kubelet/pods/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad/volumes/kubernetes.io~projected/kube-api-access-qhw5r major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c62b53e2-1a0a-43bb-a0b8-8181d53f2384/volumes/kubernetes.io~projected/kube-api-access-gx77q:{mountpoint:/var/lib/kubelet/pods/c62b53e2-1a0a-43bb-a0b8-8181d53f2384/volumes/kubernetes.io~projected/kube-api-access-gx77q major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~projected/kube-api-access-lmjf4:{mountpoint:/var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~projected/kube-api-access-lmjf4 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~projected/kube-api-access-khzkp:{mountpoint:/var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~projected/kube-api-access-khzkp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb7319d0-ad0f-4af5-b0b6-31664beab34e/volumes/kubernetes.io~projected/kube-api-access-zmk97:{mountpoint:/var/lib/kubelet/pods/cb7319d0-ad0f-4af5-b0b6-31664beab34e/volumes/kubernetes.io~projected/kube-api-access-zmk97 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~projected/kube-api-access-kdz5w:{mountpoint:/var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~projected/kube-api-access-kdz5w major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5f69512aee80c3bc856f766cf7505849d48d2c7cb175be16090907af1f6aae98/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f0f4d2fe6c53a067b06c656a18c3fac8890acca3d548ff65442c7e428297ca15/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/eafa92c545a4f7fea63ee02f9d82a860cf28009bb5ae28d7666a71fad162e271/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/fdb3781a7092d7b40f7c0c07f0c766893b3902d110954cf3eb84718353ea2818/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6c6ee7c1cb380403a09400bf756019f9b2b910e1bbfd40f7115562aea2b15ecd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/63064105b3b754caa46591fef88f7d629bcbe54e67d4694a2cdc062f1d6b5823/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/24c0bab94b3eb57364e89a2d5877e0e3c88cc745b65d221467b733e70fe42a99/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/07322ab3460c7396f5578f80ffbe735636a2cc5974d62680c24c0789af264192/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/d72496bf804a067389ca55b3890007bf76b88c97aad48a35433d06dc36c29d3d/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c9a96745a9119a53b06478e5f74e07c0239933fa138dda223f0f7dc668c2ede9/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e50c8e89ac530afc39fdafd3929e68a4049ba26a334e790c6e7cc15719bead2a/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/aaaab734a4ebcdf1c88b3cfbcd7413b036e23ceca60257ad17712cfb7090bea0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/50bbf3460e48ed95e51c06db8b3b102cd8015dff10da8088c542c20e60f0d607/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4e2d82b7751560f942eb56db394a88e0b611c2cb520405c8d5e2ba67283ad918/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ca6560c404db45c2b825b808fb0331abdfc9bc45069c65b15834be7c16e3a4cd/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1cec43a2eefa15155434f677066add8f7becccbdb15a537b67103996cc3c2c6c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7eca2cfd2cc6132f4680604fe8135751f3e419d286515aa215c504caa28d3944/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2c86f5369452611096945e2732e73d638b2ca53c1e0721e82ac82b4bc423bbaa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b20caef781facde51d210d6218b21fbb86391e41d27362e41ce04f68d8f7de04/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/038ff3dcf21680a874d26daa360144ec8504136063c8409400f95b3dca817496/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0500d0c31742e83f64b4c58173ac2bd13087dd94059fa89e625c9e0a506730f3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4ff74a3c33fe687d80a0956c6cb400b0fc6e7f5498291f03fd818af5423af831/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b68473c4bc394e7ae00d342f54c88d67fbfb0d7cd2748a79b1a6b3f2c7e27bb5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/95307dd5a858cd8e0e8a998f92c3de48c70a1a10520dc4beb396a4f9b0d572a2/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fd56ba39f4f0a09598ca54a72afcd079856ff22ebbd5e8434dd3c2c80262fd55/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f7676006153e169400dd06c4fbea00d05973f91712d5407b6fe851be84bd703d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/aea53282dbef189f31f4f7826758f8f9b6d06b0528bfde8fdb4dfb5b0f03c4da/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/5c6fe1270f31da6db1b91ad4ff47cfb1a76c4fd24cc1d8922a14ee2bbfee5731/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/da9c28beebba3917cbf2fdd2e34b21a8a6b41936630cd5fe139cae84528e4404/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c6ac053d0b69a2fb6b44520e252b465b947b6c0ec3d090faebeaa5d408ee203d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/118a8d1edca0a838c73e535d8621b7cc0a2d56ba54475ac596e2e3ccf5193400/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/76d981aef305bb5cf0528460def742f9a63fa83a5d375f100e8491811814c0a6/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/377d3aaf425f67e631d74ca49f6acc0cd705da222cc711ccc2d0e268b15b8f3e/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/4293540055bc039d228895e37aeddb1f25a85e6fa02070f983a8bcf0adfcc461/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/f8209a06cd823a01438be5683a0cde10994ec3f5a9ce2c4192fc56a75d61ed48/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/9da39ec2dbd7597252465a0372b0edf68dc952071653935b3696302b200b3173/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/f309ae1c90ab6cbb4f04cdc190c74c96018731484d0a29990a838f23d9fc255b/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/394f89a92b30765ee14725f81fba66b499d54d59a51f0d0d97abdc4ba067cee7/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/5bde0fa9e2453230b4a544bf7f66ed1d3179d293d6061ed461ca980430276405/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0153b8f47427238b406a2ec89949a61d250e8d91ae4d9f1e48bd6ee80b960d30/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/6e63a966f2a04d61a13e275aa811197b174954f15ee9fb04562ede89069c41c1/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/d38e6cac463c2cff81d20575479133f836134a459f381a3ef22a9d8a31c6f84d/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/13d52e8e777fc4742730585163420e10fccf1b3560d482b814b7782ca977cdb2/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/dff6315d79c9539ca4200772ee73a6e8bc3b8dac57c260e216f26c53e2319836/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/cc9f955701e911df60490b327a36941d3132eea50faab5593679fc00d27170d5/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/be0b85e807912dc549dc468717438be9d575035cbb66447116e6188b0383c25e/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/5bd60e03e92d868b43fe48029c9eeb9df2664d3771d385da6c7524923e51c020/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5ff52d862b812afe79558d499b9a90e2622d070325b86c293a0e5529bc85d811/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9e826e679325005feb36aa45b41fa5eeb6d6c33f4ce5ef89a23cbc0f458e47a8/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/20efed89e2fd4d5e3fb910ece4b3e24a33b514c6e29cbb0bacc971437fe85977/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/f210bb6e615906890234c242e3d33cca15643577b6c807a3cbbe9342728db084/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/fe7c980cc8e499f353359ec4ff4ff4ee366486a34c055b4194dd1b625537f692/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/642c7fbb37cb5a3d5d90f147a3b166d99ce1260e3cf38c54956c27fd2c1bed1b/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e6c01366c7dcdf6a10faf7d01a9b19c04ff6cda952c599945d606a8ba64a99ae/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0bebb94066c9f1733876bebfaf64e3da25f4c92fb0fc44a62758eabae89b9b77/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/852d76cb651d4eb53dde920af86cee7ecf1ca73f3bf9ab65ff300349e8434bdb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/706aea37322e5c1dd7e6c5c865b71dfef7b68e08cb9f9fe5d6e7330135887743/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c75acab52e232e798d296be9b7903471f4e4b1d9362d052bb274a14b1db2525b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2a0cf9894232e625e365482b510672eeade6694681d472f5e024f08a80297a30/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2f2cb194f847da5a7f13d4c32f507dcfbf5296dd060b3b8cc7c5c79a6789b973/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/f694a47b08d8ccd8893cfe5fa1197ed68fbc2445c5b377f0e596027765a5da0a/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/eb14829b569990e4303441a03ca5b52747ff6a935230f1a72120e21ffc46342e/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.675682    5066 manager.go:217] Machine: {Timestamp:2026-11-01 20:02:26.675225559 +0000 UTC m=+0.060123140 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182881 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d637e638f704c2b2d4e9ba568b24 SystemUUID:ec27d637-e638-f704-c2b2-d4e9ba568b24 BootID:dc9b979d-c3c8-4bcf-9628-7b12a15aa898 Filesystems:[{Device:/run/containers/storage/overlay-containers/c7463d6cdef03994c31407ca1c2a3829792f46cd24aa987f0eeae882fcf7ef18/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2a563d52-ad66-4e70-98e9-10266df43ddd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddf051217ddae5524e3fd804aee079e018732f59f97fa8b0b3dfe3ed7401518b/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~projected/kube-api-access-96wmh DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a51febf4b1a29a8a75b008c2c21d65d128ac099a62b58440a5a0a90539b2f2/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1495ed93-b519-468e-8dd2-e40df2c3e979/volumes/kubernetes.io~projected/kube-api-access-8w2vg DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~projected/kube-api-access-2s2pc DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~projected/kube-api-access-68tcn DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f241c0630aac7694f194c787e24b51e76f9f127ec133b926c355a2c0d458f8a6/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2782b191-652e-4551-b380-2996d67e13b5/volumes/kubernetes.io~projected/kube-api-access-wmzf5 DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/659b88a7-6607-43cf-a880-c3ee709c3862/volumes/kubernetes.io~projected/kube-api-access-hftnp DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c/volumes/kubernetes.io~projected/kube-api-access-hcvbh DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce41e940-ffc1-4f08-93c8-3abd565d89e2/volumes/kubernetes.io~projected/kube-api-access-kdz5w DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2681ec2b3ab803ae305212e509116baa625991692938112fd437ccb74975004b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33e840a4-f321-43d0-9864-5321d68fd28e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/983468a794f893fffbc368cd832f2c4507c68f7fab794d7fb60a821b165df541/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8a270bb-d842-48e6-a847-ceced3cc014a/volumes/kubernetes.io~projected/kube-api-access-khzkp DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35dd1a4d-103c-4343-a834-3479d75e59e0/volumes/kubernetes.io~projected/kube-api-access-gsx89 DeviceMajor:0 DeviceMinor:337 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc/volumes/kubernetes.io~projected/kube-api-access-mz6bb DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42486c5c-cb87-4f5e-96b3-f957803d0b75/volumes/kubernetes.io~projected/kube-api-access-q8bc6 DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/kube-api-access-dhc4c DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01ead49b82900e29becd93514cfc5d870760ee3a983e4e81d35202d274903d8d/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/067dc6af-f795-489f-820d-b81b63250a86/volumes/kubernetes.io~projected/kube-api-access-mtwb4 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7b66bc4-cb31-484b-b73f-80eaefeb3d78/volumes/kubernetes.io~projected/kube-api-access-wf9nv DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddfd74c8e5b1df18c331df00601ced6af658f87a4b1c066f0096a1621dd6bd2a/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07b921bf-2def-4eaf-8efd-a883c81f402a/volumes/kubernetes.io~projected/kube-api-access-hbqrc DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/befdc8ba-b69c-4357-a5c8-fb748a4cb03e/volumes/kubernetes.io~projected/kube-api-access-nfq55 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad/volumes/kubernetes.io~projected/kube-api-access-qhw5r DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3014c67d-90f7-41d9-8a27-1657568714cd/volumes/kubernetes.io~projected/kube-api-access-259bj DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/134ece21c66afccaf1c30583a45156f1b439484b4d4fd3a8bd5acca2e8ba670c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28e4f5e52fc8d88896e70bb5c314e1c8faee2e3d8d6eb606ec690a5e423bfcec/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0391dc-0199-4138-97d2-4db3371dda0f/volumes/kubernetes.io~projected/kube-api-access-g6rj4 DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ae353a2d97fe22708facf8ee7ac989a0cc77b7a89b94421330d59de64305c94/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9108fec3a68d35e82b006890b186df211d90efd88368acff95a4d402b5c437d1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~projected/kube-api-access-z72k9 DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/kube-api-access-59pbf DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2eb9650d0a85f8540222c161c7f1b52490ba1e36722fe5f67cfc40f739e314ec/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c34b1735-2c5a-442b-a067-2c165eb00003/volumes/kubernetes.io~projected/kube-api-access-wnn56 DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73c09b58-e615-4304-b62a-a9733d309f60/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb7319d0-ad0f-4af5-b0b6-31664beab34e/volumes/kubernetes.io~projected/kube-api-access-zmk97 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1946120d-acc5-423a-b8e8-eec897eb0bee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/395408fa-e36c-4283-82ee-613469a4af49/volumes/kubernetes.io~projected/kube-api-access-zctp6 DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08764d89e81803acbf15b502711d5be390e7b87311801e613237a92ce1bb519e/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b59ffe7045c68a7f9275d10db09271915b3d1e493fe274a4154f4d10e99ef486/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~projected/kube-api-access-lmjf4 DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1993edf4f4f194216a9bdeb53fd16cb386516068222eb3ba014f7c190bea3a37/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99d9db4b-329f-4358-b755-ae66f4260444/volumes/kubernetes.io~projected/kube-api-access-vrgx5 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f1bf3934af3d6ad3dc7e8ae4a025899fd08fddd8572bf8542571e6cb60342d6/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f29811-63a6-4c22-8b23-ad93c7277b3b/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ff9c3f-675e-4693-adb0-9bdf4308347c/volumes/kubernetes.io~projected/kube-api-access-v42mm DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/daf303282d5bcb009330442a49b148242942c8cac624a87b9dae9f8b7ea7f82c/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a4a52df-9a7b-4332-92ec-8ddc8d299bad/volumes/kubernetes.io~projected/kube-api-access-k8rst DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42007c5b6639591a59c6fb5d9930517719a9e660b292480ab61e3f0f64cce8f9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f792404c86d54c8a18bc3400c97944f85e70347d27c4d9d047b7f42c6243c55f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea8df0615d6b731ea82ca623c1d01afc3c85fba0e48d4f047983ce44940603e3/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1a60c95e6032d312c9206419e72da0548a583afee849eec3d103e509ad1b3a5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c62b53e2-1a0a-43bb-a0b8-8181d53f2384/volumes/kubernetes.io~projected/kube-api-access-gx77q DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7adb9ea9-0207-415d-91c4-d85a59b4746c/volumes/kubernetes.io~projected/kube-api-access-tsg4k DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6e450e3b78da1730235ca57f850718a01f7a82443453861b71068173deb125/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01ead49b82900e2 MacAddress:1e:62:08:75:ec:0f Speed:10000 Mtu:8901} {Name:08764d89e81803a MacAddress:de:88:b9:6d:bc:4d Speed:10000 Mtu:8901} {Name:1993edf4f4f1942 MacAddress:6e:58:72:cc:e2:5f Speed:10000 Mtu:8901} {Name:2681ec2b3ab803a MacAddress:b2:e2:55:86:1d:66 Speed:10000 Mtu:8901} {Name:28e4f5e52fc8d88 MacAddress:ce:90:76:72:10:eb Speed:10000 Mtu:8901} {Name:2eb9650d0a85f85 MacAddress:5a:73:97:e7:8f:a3 Speed:10000 Mtu:8901} {Name:3e6e450e3b78da1 MacAddress:36:53:73:6d:18:1f Speed:10000 Mtu:8901} {Name:42007c5b6639591 MacAddress:ba:35:18:2b:c9:35 Speed:10000 Mtu:8901} {Name:4710e4db1d54b07 MacAddress:a2:69:fc:e8:bd:45 Speed:10000 Mtu:8901} {Name:983468a794f893f MacAddress:12:71:92:48:79:d3 Speed:10000 Mtu:8901} {Name:9f1bf3934af3d6a MacAddress:12:3a:18:46:e4:85 Speed:10000 Mtu:8901} {Name:b59ffe7045c68a7 MacAddress:9e:db:6a:2e:e2:aa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:39:2b:4c:93:15 Speed:0 Mtu:9001} {Name:br-int MacAddress:06:6d:f2:2f:ac:79 Speed:0 Mtu:8901} {Name:c7463d6cdef0399 MacAddress:36:90:30:aa:0c:c4 Speed:10000 Mtu:8901} {Name:daf303282d5bcb0 MacAddress:52:52:e5:22:2b:c2 Speed:10000 Mtu:8901} {Name:ea8df0615d6b731 MacAddress:46:50:4d:8b:2f:cf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:39:2b:4c:93:15 Speed:0 Mtu:9001} {Name:f792404c86d54c8 MacAddress:4e:fb:d9:10:55:4c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f6:b3:4a:4b:be:30 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:86:6d:cb:fd:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676047    5066 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676135    5066 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676458    5066 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676475    5066 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-223.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676614    5066 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676623    5066 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676636    5066 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676654    5066 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676740    5066 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676894    5066 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.676988    5066 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.677019    5066 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.677028    5066 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.677036    5066 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.677044    5066 kubelet.go:405] "Adding apiserver pod source"
Nov 01 20:02:26.677078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.677054    5066 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 20:02:26.678035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.678022    5066 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 20:02:26.678035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.678035    5066 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:02:26.678544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.678520    5066 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 20:02:26.681899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.681882    5066 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 20:02:26.681899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.681896    5066 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 20:02:26.682010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.681990    5066 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 20:02:26.682180 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682169    5066 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 20:02:26.682209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682197    5066 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 20:02:26.682291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682282    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 20:02:26.682319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682299    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 20:02:26.682319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682309    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 20:02:26.682319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682316    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682322    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682327    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682332    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682338    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682345    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682351    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682360    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682368    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682390    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 20:02:26.682421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.682396    5066 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 20:02:26.685724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.685710    5066 server.go:1263] "Started kubelet"
Nov 01 20:02:26.685840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.685789    5066 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 20:02:26.685897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.685804    5066 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 20:02:26.685897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.685878    5066 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 20:02:26.686225 ip-10-0-142-223 systemd[1]: Started Kubernetes Kubelet.
Nov 01 20:02:26.686356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.686293    5066 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 20:02:26.687040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.686969    5066 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 20:02:26.689047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689027    5066 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 20:02:26.689157 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689057    5066 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 20:02:26.689261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689217    5066 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:47 +0000 UTC" deadline="2028-08-10 05:27:15.087020956 +0000 UTC"
Nov 01 20:02:26.689261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689255    5066 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 20:02:26.689261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689258    5066 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15537h24m48.397766439s"
Nov 01 20:02:26.689389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689243    5066 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 20:02:26.689389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.689279    5066 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 20:02:26.689622 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.689600    5066 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694158    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0391dc-0199-4138-97d2-4db3371dda0f" volumeName="kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694202    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4a52df-9a7b-4332-92ec-8ddc8d299bad" volumeName="kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694223    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e840a4-f321-43d0-9864-5321d68fd28e" volumeName="kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694238    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dd1a4d-103c-4343-a834-3479d75e59e0" volumeName="kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694252    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694271    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694284    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8398821f-a5b8-4ddf-a46f-e1123e5bfbfc" volumeName="kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694303    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1495ed93-b519-468e-8dd2-e40df2c3e979" volumeName="kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694315    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0391dc-0199-4138-97d2-4db3371dda0f" volumeName="kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694325    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694335    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694343    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73c09b58-e615-4304-b62a-a9733d309f60" volumeName="kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694353    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5fd4033-6a39-49fd-ad3d-f1bda1df12ad" volumeName="kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694361    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8a270bb-d842-48e6-a847-ceced3cc014a" volumeName="kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694375    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07b921bf-2def-4eaf-8efd-a883c81f402a" volumeName="kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694382    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc" volumeName="kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694401    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e840a4-f321-43d0-9864-5321d68fd28e" volumeName="kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694410    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="659b88a7-6607-43cf-a880-c3ee709c3862" volumeName="kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694418    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="659b88a7-6607-43cf-a880-c3ee709c3862" volumeName="kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694433    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73c09b58-e615-4304-b62a-a9733d309f60" volumeName="kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694445    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694460    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694473    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694486    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6" volumeName="kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694516    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694527    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694543    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694556    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d9db4b-329f-4358-b755-ae66f4260444" volumeName="kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694581    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ff9c3f-675e-4693-adb0-9bdf4308347c" volumeName="kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694593    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb7319d0-ad0f-4af5-b0b6-31664beab34e" volumeName="kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694609    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694621    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befdc8ba-b69c-4357-a5c8-fb748a4cb03e" volumeName="kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694637    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694651    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42486c5c-cb87-4f5e-96b3-f957803d0b75" volumeName="kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694663    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted" seLinuxMountContext=""
Nov 01 20:02:26.694652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694680    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73c09b58-e615-4304-b62a-a9733d309f60" volumeName="kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694693    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb9ea9-0207-415d-91c4-d85a59b4746c" volumeName="kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694708    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067dc6af-f795-489f-820d-b81b63250a86" volumeName="kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694720    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20f7ffbf-182f-4aef-a745-fd7cd7e61c1c" volumeName="kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694731    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3014c67d-90f7-41d9-8a27-1657568714cd" volumeName="kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694748    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e840a4-f321-43d0-9864-5321d68fd28e" volumeName="kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694759    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce41e940-ffc1-4f08-93c8-3abd565d89e2" volumeName="kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694775    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba" volumeName="kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694786    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0391dc-0199-4138-97d2-4db3371dda0f" volumeName="kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694800    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2782b191-652e-4551-b380-2996d67e13b5" volumeName="kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694816    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8398821f-a5b8-4ddf-a46f-e1123e5bfbfc" volumeName="kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694840    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694856    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c34b1735-2c5a-442b-a067-2c165eb00003" volumeName="kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694870    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1946120d-acc5-423a-b8e8-eec897eb0bee" volumeName="kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694885    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc" volumeName="kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694925    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694938    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694957    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb9ea9-0207-415d-91c4-d85a59b4746c" volumeName="kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694968    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d9db4b-329f-4358-b755-ae66f4260444" volumeName="kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694980    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5fd4033-6a39-49fd-ad3d-f1bda1df12ad" volumeName="kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694996    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695027    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f29811-63a6-4c22-8b23-ad93c7277b3b" volumeName="kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695042    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8a270bb-d842-48e6-a847-ceced3cc014a" volumeName="kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695054    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395408fa-e36c-4283-82ee-613469a4af49" volumeName="kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695066    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" volumeName="kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695083    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b66bc4-cb31-484b-b73f-80eaefeb3d78" volumeName="kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695094    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c34b1735-2c5a-442b-a067-2c165eb00003" volumeName="kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695110    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c34b1735-2c5a-442b-a067-2c165eb00003" volumeName="kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695123    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce41e940-ffc1-4f08-93c8-3abd565d89e2" volumeName="kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695137    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1495ed93-b519-468e-8dd2-e40df2c3e979" volumeName="kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695152    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95635b7c-a032-45a4-9999-ef2a08af06e2" volumeName="kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695164    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067dc6af-f795-489f-820d-b81b63250a86" volumeName="kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695180    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6" volumeName="kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695193    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73c09b58-e615-4304-b62a-a9733d309f60" volumeName="kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695206    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb9ea9-0207-415d-91c4-d85a59b4746c" volumeName="kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695222    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d9db4b-329f-4358-b755-ae66f4260444" volumeName="kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695234    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f29811-63a6-4c22-8b23-ad93c7277b3b" volumeName="kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695249    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb7319d0-ad0f-4af5-b0b6-31664beab34e" volumeName="kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695260    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce41e940-ffc1-4f08-93c8-3abd565d89e2" volumeName="kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695271    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4a52df-9a7b-4332-92ec-8ddc8d299bad" volumeName="kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695286    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1495ed93-b519-468e-8dd2-e40df2c3e979" volumeName="kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695298    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42486c5c-cb87-4f5e-96b3-f957803d0b75" volumeName="kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695313    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c62b53e2-1a0a-43bb-a0b8-8181d53f2384" volumeName="kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695324    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f29811-63a6-4c22-8b23-ad93c7277b3b" volumeName="kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695336    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6" volumeName="kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695353    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a563d52-ad66-4e70-98e9-10266df43ddd" volumeName="kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695365    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="659b88a7-6607-43cf-a880-c3ee709c3862" volumeName="kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695381    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73c09b58-e615-4304-b62a-a9733d309f60" volumeName="kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695391    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb9ea9-0207-415d-91c4-d85a59b4746c" volumeName="kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695405    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c34b1735-2c5a-442b-a067-2c165eb00003" volumeName="kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695419    5066 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce41e940-ffc1-4f08-93c8-3abd565d89e2" volumeName="kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert" seLinuxMountContext=""
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695429    5066 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695436    5066 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.694787    5066 factory.go:55] Registering systemd factory
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695686    5066 factory.go:223] Registration of the systemd container factory successfully
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695889    5066 factory.go:153] Registering CRI-O factory
Nov 01 20:02:26.696209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.695901    5066 factory.go:223] Registration of the crio container factory successfully
Nov 01 20:02:26.698183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.696462    5066 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 20:02:26.698183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.696485    5066 factory.go:103] Registering Raw factory
Nov 01 20:02:26.698183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.696526    5066 manager.go:1199] Started watching for new ooms in manager
Nov 01 20:02:26.698183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.696852    5066 manager.go:319] Starting recovery of all containers
Nov 01 20:02:26.706078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.706043    5066 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 20:02:26.714969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.714947    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:26.715053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.715000    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:26.722865 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.722841    5066 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 20:02:26.722944 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.722871    5066 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 20:02:26.722944 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.722896    5066 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 20:02:26.723010 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.722938    5066 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 20:02:26.739318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.739292    5066 generic.go:356] "Generic (PLEG): container finished" podID="61bc8eba7840edd90e06bc583d329627" containerID="edae60eccbe3c39f4ad5cf478257de7381265fdf58ce08dbaeb7bdc2d8a00d40" exitCode=0
Nov 01 20:02:26.745214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.745195    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:26.745293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.745221    5066 generic.go:356] "Generic (PLEG): container finished" podID="ce41e940-ffc1-4f08-93c8-3abd565d89e2" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14" exitCode=255
Nov 01 20:02:26.752717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752691    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="b03f48ff465fd07c0465300e85787f57e5937073ea8e0a6f35fce10d0cb64ae2" exitCode=0
Nov 01 20:02:26.752717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752715    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="8a970286497806e9f365bbf40e1963ee7838da07205e6ff7a5ed33c7cd352384" exitCode=0
Nov 01 20:02:26.752843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752721    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="b1cffe778c96aa2d514ee743eae33c57c34039c59236ad3fc778e5550e31fdd2" exitCode=0
Nov 01 20:02:26.752843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752726    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="a3119840144feba6cbe89f0eaeeea199ca9e39dab33285607190477ba7ce2550" exitCode=0
Nov 01 20:02:26.752843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752731    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="77be49e885e62f9bf5ee8467145e99cb104afe7315741ce9608fcbc7ed9a783e" exitCode=0
Nov 01 20:02:26.752843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.752737    5066 generic.go:356] "Generic (PLEG): container finished" podID="7adb9ea9-0207-415d-91c4-d85a59b4746c" containerID="291585c6b2951cb36cef8d8b1391e6bbe6c2f0c2d457ba48fe5634ed0b28f86d" exitCode=0
Nov 01 20:02:26.789779 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.789755    5066 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:02:26.792013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.791998    5066 manager.go:324] Recovery completed
Nov 01 20:02:26.811254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.811232    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.812431 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.812410    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.812518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.812441    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.812518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.812452    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.814971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.814956    5066 cpu_manager.go:226] "Starting" policy="none"
Nov 01 20:02:26.814971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.814969    5066 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 20:02:26.815067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.814971    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.815067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.814985    5066 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 20:02:26.815133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815093    5066 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 20:02:26.815133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815103    5066 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 20:02:26.815133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815120    5066 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 20:02:26.815133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815126    5066 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 20:02:26.815133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815130    5066 policy_none.go:50] "Start"
Nov 01 20:02:26.815264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815138    5066 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 20:02:26.815264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815146    5066 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 20:02:26.815338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815265    5066 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 20:02:26.815338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815272    5066 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 20:02:26.815338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815277    5066 policy_none.go:44] "Start"
Nov 01 20:02:26.815761 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815748    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.815822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815772    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.815822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.815782    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.819617 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.819591    5066 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 20:02:26.822688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.822674    5066 manager.go:341] "Starting Device Plugin manager"
Nov 01 20:02:26.822768 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.822697    5066 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 20:02:26.822768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.822705    5066 server.go:87] "Starting device plugin registration server"
Nov 01 20:02:26.822937 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.822925    5066 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 20:02:26.822989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.822936    5066 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 20:02:26.823049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823024    5066 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 20:02:26.823102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823058    5066 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"]
Nov 01 20:02:26.823176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823143    5066 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 20:02:26.823227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823183    5066 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 20:02:26.823227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823191    5066 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 20:02:26.823227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823184    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.823370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.823278    5066 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:02:26.823174278Z","UUID":"ec28ca87-a9ee-4020-afe0-10ae4049df25","Handler":null,"Name":"","Endpoint":""}
Nov 01 20:02:26.824274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824239    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.824274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824267    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.824477 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824281    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.824686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824665    5066 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 20:02:26.824779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824701    5066 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 20:02:26.824779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824667    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.825009 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.824871    5066 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 20:02:26.825009 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.824921    5066 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-223.us-west-1.compute.internal\" not found"
Nov 01 20:02:26.825009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.824766    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.825979    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.826020    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.826040    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.826215    5066 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.826289    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.826314    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.826693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.826335    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.827311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827034    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerStarted","Data":"820c98c52d27c6d0dcfcac309dc7aad1a6f7d079757d857e5f50ad3b67416c13"}
Nov 01 20:02:26.827311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827094    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerDied","Data":"edae60eccbe3c39f4ad5cf478257de7381265fdf58ce08dbaeb7bdc2d8a00d40"}
Nov 01 20:02:26.827311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827129    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"61bc8eba7840edd90e06bc583d329627","Type":"ContainerStarted","Data":"4a0ab4aacacf9e87684ef66ecceb325c80bb31de7ee27a3f2f1165db36cf4002"}
Nov 01 20:02:26.827311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827262    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"88d138da692bc0e2c453d9d2e3886bfc","Type":"ContainerStarted","Data":"d7ab2ccd5f2644be416c688e4b58f50750a2bfd67497ae00ac4cf76cc9cad0ea"}
Nov 01 20:02:26.827311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827294    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal" event={"ID":"88d138da692bc0e2c453d9d2e3886bfc","Type":"ContainerStarted","Data":"4d93faf7e88080bfdbb2d5d90486282dd9dd1199ac4e950d5f2364bd0aee0ef8"}
Nov 01 20:02:26.827604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.827425    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.829072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.828765    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.829072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.828801    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.829072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.828820    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.829072 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.829063    5066 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-223.us-west-1.compute.internal\" not found" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.884717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.884689    5066 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 20:02:26.885437 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.885421    5066 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:02:26.886073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.886048    5066 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:02:26.886190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.886123    5066 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:02:26.890198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.890184    5066 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.923346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.923320    5066 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 20:02:26.924461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.924440    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 20:02:26.924548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.924476    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 20:02:26.924548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.924487    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 20:02:26.924681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.924592    5066 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.935792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.935726    5066 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 20:02:26.948683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.948659    5066 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:02:26.948772 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:26.948717    5066 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.948772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.948734    5066 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.984920 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.984895    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.985027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.984931    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:26.985027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:26.984951    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.037572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.037550    5066 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.037650 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.037609    5066 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.038231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.038096    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 20:02:27.038231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.038132    5066 setters.go:546] "Node became not ready" node="ip-10-0-142-223.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T20:02:27Z","lastTransitionTime":"2026-11-01T20:02:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 20:02:27.045232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.045211    5066 nodeinfomanager.go:471] existing CSINode "ip-10-0-142-223.us-west-1.compute.internal" is owned by different node (oldNodeID="af8c6c41-31f6-403a-a3a0-d7dc498e89c8", newNodeID=""), cleaning up...
Nov 01 20:02:27.051053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.051039    5066 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:02:27.051115 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.051078    5066 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085216    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085256    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085278    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085224    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085338    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61bc8eba7840edd90e06bc583d329627-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"61bc8eba7840edd90e06bc583d329627\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.085436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.085313    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d138da692bc0e2c453d9d2e3886bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" (UID: \"88d138da692bc0e2c453d9d2e3886bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.679647 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.679552    5066 apiserver.go:52] "Watching apiserver"
Nov 01 20:02:27.777438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.777412    5066 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.831954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.831920    5066 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 20:02:27.832637 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.832613    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn","openshift-dns/dns-default-94r67","openshift-network-diagnostics/network-check-target-q4r47","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw","openshift-network-operator/iptables-alerter-sjn27","openshift-image-registry/node-ca-zq4jc","openshift-insights/periodic-gathering-4d7xr-lr6rw","openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r","openshift-multus/network-metrics-daemon-hqhjj","openshift-image-registry/image-registry-56cd6f9db5-6m4r2","kube-system/kas-connection-checker-685895c7c9-qgkq7","openshift-cluster-node-tuning-operator/tuned-dvx8k","openshift-ovn-kubernetes/ovnkube-node-sh57k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v","openshift-ingress-canary/ingress-canary-8dbcd","openshift-multus/multus-additional-cni-plugins-wrz6h","openshift-multus/multus-cmcvj","openshift-network-console/networking-console-plugin-86c478bb88-2kwmf","openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs","openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh","openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w","openshift-dns/node-resolver-fm8ml","openshift-insights/insights-runtime-extractor-sr5gb","openshift-service-ca/service-ca-5fffc97bdf-tjssh","kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal","openshift-insights/insights-operator-7c4986d449-m49w5","kube-system/konnectivity-agent-ktldr","openshift-console-operator/console-operator-7cd56b8dbc-dlrcw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-223.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-2zq5k","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq","kube-system/kas-connection-checker-685895c7c9-ckjjq","openshift-image-registry/image-registry-5d7b96b89b-7xj6x","openshift-ingress/router-default-646fc8f47-znjf4","openshift-network-diagnostics/network-check-source-6f78d59f99-xl792","kube-system/global-pull-secret-syncer-nn89g"]
Nov 01 20:02:27.833830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.833814    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:27.834120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.834090    5066 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 20:02:27.834280 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.834259    5066 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal"
Nov 01 20:02:27.835039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.835024    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:27.836117 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.836094    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.840235 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.840215    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:27.840636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.840620    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:27.843527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.843488    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.846720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.846704    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:27.847404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.847389    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:27.848572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.848560    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.885262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.885239    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.886007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.885988    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.886075 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886032    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.886376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886363    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.886443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886392    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 20:02:27.886606 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886590    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 20:02:27.886767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886751    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.886847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886802    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.886847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886843    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.886945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886890    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 20:02:27.886994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.886971    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.893572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893535    5066 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 20:02:27.893842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893823    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.893916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893874    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9nv\" (UniqueName: \"kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:27.893916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893908    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx77q\" (UniqueName: \"kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q\") pod \"volume-data-source-validator-58c9f5f46d-x8tnw\" (UID: \"c62b53e2-1a0a-43bb-a0b8-8181d53f2384\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:27.894003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893940    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:27.894003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893969    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.894087 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.893999    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.894087 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894046    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.894087 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894074    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.894225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894106    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdz5w\" (UniqueName: \"kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.894225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894138    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2pc\" (UniqueName: \"kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.894225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894162    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.894225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894190    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.894225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894217    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.894521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894247    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.894521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894314    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.894521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894376    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.894521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894459    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:27.894712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894568    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.894712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894603    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42486c5c-cb87-4f5e-96b3-f957803d0b75-data-volume\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.894810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894715    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.894810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894800    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.894891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894875    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.894995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.894965    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.895309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895290    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfq55\" (UniqueName: \"kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55\") pod \"kas-connection-checker-685895c7c9-2zq5k\" (UID: \"befdc8ba-b69c-4357-a5c8-fb748a4cb03e\") " pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:27.895371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895315    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42mm\" (UniqueName: \"kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.895448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895435    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.895513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895461    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.895513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895478    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvbh\" (UniqueName: \"kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh\") pod \"network-check-source-6f78d59f99-xl792\" (UID: \"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:27.895613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895512    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:27.895613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895536    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:27.895613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895562    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.895613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895588    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.895785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895638    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.895785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895671    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.895785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895697    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.895785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895725    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.895785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895762    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.896003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895839    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rst\" (UniqueName: \"kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.896003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895872    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:27.896003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.895898    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:27.896192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896128    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.896192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896151    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk97\" (UniqueName: \"kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:27.896192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896173    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896198    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896221    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896244    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896267    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896291    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.896319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896315    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bc6\" (UniqueName: \"kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896340    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896390    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctp6\" (UniqueName: \"kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6\") pod \"kas-connection-checker-685895c7c9-qgkq7\" (UID: \"395408fa-e36c-4283-82ee-613469a4af49\") " pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896413    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896440    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896464    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.896608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896506    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.896869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896611    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-config\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.896869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896760    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.896869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896806    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:27.896869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896841    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.896869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896845    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-snapshots\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.897076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896903    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.897076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896936    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.897076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.896977    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.897216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897076    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.897216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897127    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.897216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897168    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1946120d-acc5-423a-b8e8-eec897eb0bee-tmp\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.897216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897198    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:27.897373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897223    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.897373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897247    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.897373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897302    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.897373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897320    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.897373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.897338    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftnp\" (UniqueName: \"kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.935763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.935690    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 20:02:27.935868 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.935815    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 20:02:27.935985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.935969    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.936043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936010    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 20:02:27.936043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936011    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 20:02:27.936043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936034    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.936182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936017    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.936182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936085    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.936182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936126    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.936309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936204    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.936309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936292    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mr6jl\""
Nov 01 20:02:27.936390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936355    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 20:02:27.936390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936356    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4jvr\""
Nov 01 20:02:27.936486 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936475    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936556    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936615    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-247ns\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936622    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wmgh6\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936649    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936661    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936656    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936689    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce41e940-ffc1-4f08-93c8-3abd565d89e2-trusted-ca\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.936737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936708    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7gfgj\""
Nov 01 20:02:27.937030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936786    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 20:02:27.937030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936841    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dmdkt\""
Nov 01 20:02:27.937030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936873    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 20:02:27.937030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.936970    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ftcsj\""
Nov 01 20:02:27.937182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.937167    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1946120d-acc5-423a-b8e8-eec897eb0bee-serving-cert\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.937425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.937409    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.946729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.946703    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce41e940-ffc1-4f08-93c8-3abd565d89e2-serving-cert\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.946729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.946716    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/659b88a7-6607-43cf-a880-c3ee709c3862-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.946847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.946747    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1946120d-acc5-423a-b8e8-eec897eb0bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.946847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.946782    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/659b88a7-6607-43cf-a880-c3ee709c3862-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.949718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.949695    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.949844 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.949745    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.949934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.949903    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f22l2\""
Nov 01 20:02:27.949934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.949909    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 20:02:27.950168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950150    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 20:02:27.950238 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950187    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 20:02:27.950238 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950229    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cc7fv\""
Nov 01 20:02:27.950330 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950190    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 20:02:27.950330 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950280    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.950425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950385    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 20:02:27.950425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950413    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 20:02:27.950546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950426    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.950546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950437    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.950546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950445    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 20:02:27.950546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950474    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.950546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950541    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950583    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950597    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950618    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950582    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950675    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950711    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950779    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.950808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950791    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950814    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950837    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950864    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950880    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950911    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950936    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.950988    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951033    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951073    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951096    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951122    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951130    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951153    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951180    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951247    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951315    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951326    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fqldw\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951355    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951405    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951424    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951433    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951405    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951475    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951482    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951358    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951524    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951534    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951564    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951577    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951604    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951609    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\""
Nov 01 20:02:27.951689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951611    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951983    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.951999    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952028    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952045    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p45lr\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952086    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952094    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952129    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952244    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952288    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952327    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952346    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952384    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952246    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952447    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\""
Nov 01 20:02:27.952465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952461    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bwz74\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952482    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952542    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952557    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952621    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952628    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952648    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 20:02:27.953092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.952701    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2mhc5\""
Nov 01 20:02:27.953390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.953155    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctp6\" (UniqueName: \"kubernetes.io/projected/395408fa-e36c-4283-82ee-613469a4af49-kube-api-access-zctp6\") pod \"kas-connection-checker-685895c7c9-qgkq7\" (UID: \"395408fa-e36c-4283-82ee-613469a4af49\") " pod="kube-system/kas-connection-checker-685895c7c9-qgkq7"
Nov 01 20:02:27.953436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.953387    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rst\" (UniqueName: \"kubernetes.io/projected/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-kube-api-access-k8rst\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.953481 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.953433    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx77q\" (UniqueName: \"kubernetes.io/projected/c62b53e2-1a0a-43bb-a0b8-8181d53f2384-kube-api-access-gx77q\") pod \"volume-data-source-validator-58c9f5f46d-x8tnw\" (UID: \"c62b53e2-1a0a-43bb-a0b8-8181d53f2384\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x8tnw"
Nov 01 20:02:27.953849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.953830    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfq55\" (UniqueName: \"kubernetes.io/projected/befdc8ba-b69c-4357-a5c8-fb748a4cb03e-kube-api-access-nfq55\") pod \"kas-connection-checker-685895c7c9-2zq5k\" (UID: \"befdc8ba-b69c-4357-a5c8-fb748a4cb03e\") " pod="kube-system/kas-connection-checker-685895c7c9-2zq5k"
Nov 01 20:02:27.953944 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.953919    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzf5\" (UniqueName: \"kubernetes.io/projected/2782b191-652e-4551-b380-2996d67e13b5-kube-api-access-wmzf5\") pod \"network-check-target-q4r47\" (UID: \"2782b191-652e-4551-b380-2996d67e13b5\") " pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:27.954377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.954357    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.954665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.954641    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdz5w\" (UniqueName: \"kubernetes.io/projected/ce41e940-ffc1-4f08-93c8-3abd565d89e2-kube-api-access-kdz5w\") pod \"console-operator-7cd56b8dbc-dlrcw\" (UID: \"ce41e940-ffc1-4f08-93c8-3abd565d89e2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:27.955045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955028    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2pc\" (UniqueName: \"kubernetes.io/projected/1946120d-acc5-423a-b8e8-eec897eb0bee-kube-api-access-2s2pc\") pod \"insights-operator-7c4986d449-m49w5\" (UID: \"1946120d-acc5-423a-b8e8-eec897eb0bee\") " pod="openshift-insights/insights-operator-7c4986d449-m49w5"
Nov 01 20:02:27.955181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955163    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9nv\" (UniqueName: \"kubernetes.io/projected/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-kube-api-access-wf9nv\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:27.955594 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955577    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvbh\" (UniqueName: \"kubernetes.io/projected/20f7ffbf-182f-4aef-a745-fd7cd7e61c1c-kube-api-access-hcvbh\") pod \"network-check-source-6f78d59f99-xl792\" (UID: \"20f7ffbf-182f-4aef-a745-fd7cd7e61c1c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xl792"
Nov 01 20:02:27.955698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955606    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk97\" (UniqueName: \"kubernetes.io/projected/cb7319d0-ad0f-4af5-b0b6-31664beab34e-kube-api-access-zmk97\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:27.955762 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955740    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.955859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.955843    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb7319d0-ad0f-4af5-b0b6-31664beab34e-serviceca\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:27.956120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956102    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-daemon-config\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.956188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956113    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42mm\" (UniqueName: \"kubernetes.io/projected/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kube-api-access-v42mm\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.956304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956287    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:27.956401 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956383    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.956559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956543    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-agent-certs\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:27.956608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956559    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8398821f-a5b8-4ddf-a46f-e1123e5bfbfc-konnectivity-ca\") pod \"konnectivity-agent-ktldr\" (UID: \"8398821f-a5b8-4ddf-a46f-e1123e5bfbfc\") " pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:27.956797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956778    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0391dc-0199-4138-97d2-4db3371dda0f-cni-binary-copy\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.956897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956866    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.956947 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.956934    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.957024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.957010    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.958055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.958033    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a563d52-ad66-4e70-98e9-10266df43ddd-original-pull-secret\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.984758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.984735    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftnp\" (UniqueName: \"kubernetes.io/projected/659b88a7-6607-43cf-a880-c3ee709c3862-kube-api-access-hftnp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq\" (UID: \"659b88a7-6607-43cf-a880-c3ee709c3862\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq"
Nov 01 20:02:27.985308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.985289    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:27.985399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.985375    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bc6\" (UniqueName: \"kubernetes.io/projected/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-api-access-q8bc6\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.997600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997572    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:27.997600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997607    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997622    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997640    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997714    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997746    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-etc-tuned\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997749    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.997791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997786    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997806    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-sys-fs\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997812    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rj4\" (UniqueName: \"kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997835    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259bj\" (UniqueName: \"kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj\") pod \"migrator-66df5d496b-996rs\" (UID: \"3014c67d-90f7-41d9-8a27-1657568714cd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997858    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997863    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997880    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997905    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997931    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997960    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997969    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-stats-auth\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.997984    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998007    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998032    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgx5\" (UniqueName: \"kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998034    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-dbus\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998054    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998118    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6bb\" (UniqueName: \"kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:27.998169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998153    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998180    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998205    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998238    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998268    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998301    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998324    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998339    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998347    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998383    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998402    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42486c5c-cb87-4f5e-96b3-f957803d0b75-crio-socket\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998407    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998431    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998457    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998484    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998524    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2vg\" (UniqueName: \"kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998543    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998601    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-system-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998656    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-cni-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.998726    5066 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998791    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.998821    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs podName:b7b66bc4-cb31-484b-b73f-80eaefeb3d78 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.498799238 +0000 UTC m=+1.883696826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs") pod "network-metrics-daemon-hqhjj" (UID: "b7b66bc4-cb31-484b-b73f-80eaefeb3d78") : secret "metrics-daemon-secret" not found
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998906    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-bin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.998965    5066 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:27.999009    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.498996833 +0000 UTC m=+1.883894404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999061    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-socket-dir-parent\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998548    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-iptables-alerter-script\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999111    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999142    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:27.999172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999169    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999273    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a563d52-ad66-4e70-98e9-10266df43ddd-kubelet-config\") pod \"global-pull-secret-syncer-nn89g\" (UID: \"2a563d52-ad66-4e70-98e9-10266df43ddd\") " pod="kube-system/global-pull-secret-syncer-nn89g"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999834    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999421    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.998847    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a270bb-d842-48e6-a847-ceced3cc014a-cert\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999522    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000247    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:27.999383    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-hostroot\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000334    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000361    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000385    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000427    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000455    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000483    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000560    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000588    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000615    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000640    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000684    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000715    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.000751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000739    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000763    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000802    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.000821    5066 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000827    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx89\" (UniqueName: \"kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89\") pod \"migrator-66df5d496b-c7hsh\" (UID: \"35dd1a4d-103c-4343-a834-3479d75e59e0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000853    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000889    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw5r\" (UniqueName: \"kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.000931    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.500910662 +0000 UTC m=+1.885808275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000959    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.000988    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001013    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjf4\" (UniqueName: \"kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001037    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001070    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001105    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001123    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb7319d0-ad0f-4af5-b0b6-31664beab34e-host\") pod \"node-ca-zq4jc\" (UID: \"cb7319d0-ad0f-4af5-b0b6-31664beab34e\") " pod="openshift-image-registry/node-ca-zq4jc"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001127    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001178    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001214    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001287    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-multus-conf-dir\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001414    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001453    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001532    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001560    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001594    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-config\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001583    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001641    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-kubelet\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001644    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwb4\" (UniqueName: \"kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001663    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e840a4-f321-43d0-9864-5321d68fd28e-config\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001686    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001702    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067dc6af-f795-489f-820d-b81b63250a86-tmp-dir\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001716    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001804    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1495ed93-b519-468e-8dd2-e40df2c3e979-tmp-dir\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.001817    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.001830    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.001892    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.501876552 +0000 UTC m=+1.886774135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001888    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.001923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001928    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.001953    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002005    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-ovnkube-script-lib\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002008    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002061    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002079    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-default-certificate\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002088    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002110    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002140    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-socket-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002139    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72k9\" (UniqueName: \"kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002262    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-cabundle\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002330    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-cnibin\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002459    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002178    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002666    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002796    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002810    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e840a4-f321-43d0-9864-5321d68fd28e-serving-cert\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002824    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002850    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002907    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.002961    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99d9db4b-329f-4358-b755-ae66f4260444-tmp\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003031    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003064    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003109    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003141    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wmh\" (UniqueName: \"kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003188    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003232    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73c09b58-e615-4304-b62a-a9733d309f60-ovn-node-metrics-cert\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.003256    5066 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.003275    5066 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003281    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-var-lib-cni-multus\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.003300    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.503285532 +0000 UTC m=+1.888183108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.003393    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.503378249 +0000 UTC m=+1.888275837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003423    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003471    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003550    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003555    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1495ed93-b519-468e-8dd2-e40df2c3e979-config-volume\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003606    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003609    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-multus-certs\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003614    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-netns\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003641    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tcn\" (UniqueName: \"kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003683    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg4k\" (UniqueName: \"kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003775    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73c09b58-e615-4304-b62a-a9733d309f60-env-overrides\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003803    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003850    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-device-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.004267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.003926    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.011458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011328    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.011458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011395    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.011458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011393    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.011671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011477    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzkp\" (UniqueName: \"kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:28.011671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011556    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.011671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011612    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.011671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011645    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.011837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011705    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.011837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011738    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.011837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.011770    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012045    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-registration-dir\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012078    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5ff9c3f-675e-4693-adb0-9bdf4308347c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwq2v\" (UID: \"b5ff9c3f-675e-4693-adb0-9bdf4308347c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwq2v"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012286    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7adb9ea9-0207-415d-91c4-d85a59b4746c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012460    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012528    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012581    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012614    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012675    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012704    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.012738    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqrc\" (UniqueName: \"kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc\") pod \"kas-connection-checker-685895c7c9-ckjjq\" (UID: \"07b921bf-2def-4eaf-8efd-a883c81f402a\") " pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.013012    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6f29811-63a6-4c22-8b23-ad93c7277b3b-signing-key\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.015112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.013057    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-host-run-k8s-cni-cncf-io\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.053130 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.053095    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259bj\" (UniqueName: \"kubernetes.io/projected/3014c67d-90f7-41d9-8a27-1657568714cd-kube-api-access-259bj\") pod \"migrator-66df5d496b-996rs\" (UID: \"3014c67d-90f7-41d9-8a27-1657568714cd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-996rs"
Nov 01 20:02:28.053265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.053148    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rj4\" (UniqueName: \"kubernetes.io/projected/0a0391dc-0199-4138-97d2-4db3371dda0f-kube-api-access-g6rj4\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.053827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.053788    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.053950 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.053880    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6bb\" (UniqueName: \"kubernetes.io/projected/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-kube-api-access-mz6bb\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:28.054060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.054043    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2vg\" (UniqueName: \"kubernetes.io/projected/1495ed93-b519-468e-8dd2-e40df2c3e979-kube-api-access-8w2vg\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.054374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.054355    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgx5\" (UniqueName: \"kubernetes.io/projected/99d9db4b-329f-4358-b755-ae66f4260444-kube-api-access-vrgx5\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.056220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.056197    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw5r\" (UniqueName: \"kubernetes.io/projected/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-kube-api-access-qhw5r\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.057127 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.057046    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjf4\" (UniqueName: \"kubernetes.io/projected/c6f29811-63a6-4c22-8b23-ad93c7277b3b-kube-api-access-lmjf4\") pod \"service-ca-5fffc97bdf-tjssh\" (UID: \"c6f29811-63a6-4c22-8b23-ad93c7277b3b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tjssh"
Nov 01 20:02:28.057270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.057253    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72k9\" (UniqueName: \"kubernetes.io/projected/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-kube-api-access-z72k9\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.057481 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.057463    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"periodic-gathering-4d7xr-lr6rw\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") " pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:02:28.058603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.058582    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx89\" (UniqueName: \"kubernetes.io/projected/35dd1a4d-103c-4343-a834-3479d75e59e0-kube-api-access-gsx89\") pod \"migrator-66df5d496b-c7hsh\" (UID: \"35dd1a4d-103c-4343-a834-3479d75e59e0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-c7hsh"
Nov 01 20:02:28.058692 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.058629    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwb4\" (UniqueName: \"kubernetes.io/projected/067dc6af-f795-489f-820d-b81b63250a86-kube-api-access-mtwb4\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.058902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.058855    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wmh\" (UniqueName: \"kubernetes.io/projected/33e840a4-f321-43d0-9864-5321d68fd28e-kube-api-access-96wmh\") pod \"service-ca-operator-544d749c8c-2f66w\" (UID: \"33e840a4-f321-43d0-9864-5321d68fd28e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w"
Nov 01 20:02:28.058966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.058954    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.059419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.059398    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tcn\" (UniqueName: \"kubernetes.io/projected/73c09b58-e615-4304-b62a-a9733d309f60-kube-api-access-68tcn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.059599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.059581    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg4k\" (UniqueName: \"kubernetes.io/projected/7adb9ea9-0207-415d-91c4-d85a59b4746c-kube-api-access-tsg4k\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.066469 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.066452    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqrc\" (UniqueName: \"kubernetes.io/projected/07b921bf-2def-4eaf-8efd-a883c81f402a-kube-api-access-hbqrc\") pod \"kas-connection-checker-685895c7c9-ckjjq\" (UID: \"07b921bf-2def-4eaf-8efd-a883c81f402a\") " pod="kube-system/kas-connection-checker-685895c7c9-ckjjq"
Nov 01 20:02:28.066647 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.066629    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzkp\" (UniqueName: \"kubernetes.io/projected/c8a270bb-d842-48e6-a847-ceced3cc014a-kube-api-access-khzkp\") pod \"ingress-canary-8dbcd\" (UID: \"c8a270bb-d842-48e6-a847-ceced3cc014a\") " pod="openshift-ingress-canary/ingress-canary-8dbcd"
Nov 01 20:02:28.113253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113221    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113261    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113286    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113310    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113336    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113353    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-cnibin\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113368    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0391dc-0199-4138-97d2-4db3371dda0f-etc-kubernetes\") pod \"multus-cmcvj\" (UID: \"0a0391dc-0199-4138-97d2-4db3371dda0f\") " pod="openshift-multus/multus-cmcvj"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113358    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113402    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113353    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-systemd\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113418    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113444    5066 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113453    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113477    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113489    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113517    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.613483122 +0000 UTC m=+1.998380690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113522    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113542    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-run\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113545    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113427    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-ovn\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113570    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.613554219 +0000 UTC m=+1.998451810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113572    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/067dc6af-f795-489f-820d-b81b63250a86-hosts-file\") pod \"node-resolver-fm8ml\" (UID: \"067dc6af-f795-489f-820d-b81b63250a86\") " pod="openshift-dns/node-resolver-fm8ml"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113586    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-sys\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113625    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113644    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113659    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113664    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113702    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113722    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc-host-slash\") pod \"iptables-alerter-sjn27\" (UID: \"19535b6d-3a55-4cd3-b3f5-d0f8331fa8dc\") " pod="openshift-network-operator/iptables-alerter-sjn27"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113727    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113746    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113746    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113706    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-host\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113782    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113788    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113800    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113830    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113851    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113875    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113910    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-run-systemd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113909    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113931    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-log-socket\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.113916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113876    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-kubernetes\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113952    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.113969    5066 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.113978    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114002    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.114011    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.613998179 +0000 UTC m=+1.998895765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114017    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-run-netns\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114039    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114039    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysconfig\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114080    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114089    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-modprobe-d\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114105    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114135    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-var-lib-kubelet\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114143    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114160    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-kubelet\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114171    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114171    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-systemd-units\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114191    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-etc-sysctl-conf\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114210    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-var-lib-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114196    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114228    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-node-log\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114248    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114272    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114313    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114329    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-slash\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114336    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114355    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-etc-openvswitch\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114360    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-service-ca-bundle\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114374    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114390    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99d9db4b-329f-4358-b755-ae66f4260444-lib-modules\") pod \"tuned-dvx8k\" (UID: \"99d9db4b-329f-4358-b755-ae66f4260444\") " pod="openshift-cluster-node-tuning-operator/tuned-dvx8k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114404    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7adb9ea9-0207-415d-91c4-d85a59b4746c-system-cni-dir\") pod \"multus-additional-cni-plugins-wrz6h\" (UID: \"7adb9ea9-0207-415d-91c4-d85a59b4746c\") " pod="openshift-multus/multus-additional-cni-plugins-wrz6h"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114395    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114429    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-bin\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114419    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73c09b58-e615-4304-b62a-a9733d309f60-host-cni-netd\") pod \"ovnkube-node-sh57k\" (UID: \"73c09b58-e615-4304-b62a-a9733d309f60\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.114445    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.114528    5066 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:28.115021 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.114571    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:28.614558734 +0000 UTC m=+1.999456320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:28.134987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.134964    5066 scope.go:122] "RemoveContainer" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"
Nov 01 20:02:28.233784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.233758    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:28.518187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518153    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:28.518187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518189    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:28.518407 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518275    5066 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:28.518407 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518286    5066 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:28.518407 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518336    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.518325155 +0000 UTC m=+2.903222740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:28.518407 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518352    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.518346563 +0000 UTC m=+2.903244131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:28.518407 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518382    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:28.518604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518417    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:28.518604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518468    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:28.518604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518528    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:28.518604 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518583    5066 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:28.518604 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518598    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:28.518757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518603    5066 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 20:02:28.518757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518626    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.518612034 +0000 UTC m=+2.903509616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:28.518757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518667    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert podName:d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.518651938 +0000 UTC m=+2.903549520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2kwmf" (UID: "d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba") : secret "networking-console-plugin-cert" not found
Nov 01 20:02:28.518757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518605    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:28.518757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.518699    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.518692902 +0000 UTC m=+2.903590471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:28.518920 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.518817    5066 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 20:02:28.520966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.520940    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b66bc4-cb31-484b-b73f-80eaefeb3d78-metrics-certs\") pod \"network-metrics-daemon-hqhjj\" (UID: \"b7b66bc4-cb31-484b-b73f-80eaefeb3d78\") " pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:28.619829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.619790    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:28.620006 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.619841    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:28.620006 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.619899    5066 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:28.620006 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.619927    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:28.620006 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.619962    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.619941347 +0000 UTC m=+3.004838929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:28.620006 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620001    5066 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620012    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620029    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620045    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.62003164 +0000 UTC m=+3.004929228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.620043    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620071    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.620057648 +0000 UTC m=+3.004955232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620082    5066 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:28.620185 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.620117    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.620106372 +0000 UTC m=+3.005003955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:28.748006 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.747977    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hqhjj"
Nov 01 20:02:28.782822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.782760    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:02:28.783236 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.783218    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/1.log"
Nov 01 20:02:28.783283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.783260    5066 generic.go:356] "Generic (PLEG): container finished" podID="ce41e940-ffc1-4f08-93c8-3abd565d89e2" containerID="0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2" exitCode=255
Nov 01 20:02:28.783356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.783331    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" event={"ID":"ce41e940-ffc1-4f08-93c8-3abd565d89e2","Type":"ContainerDied","Data":"0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2"}
Nov 01 20:02:28.783408 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.783395    5066 scope.go:122] "RemoveContainer" containerID="e1af929be0d5a50269a1150e0421c6ab9380b63b374f12e36ead4accde0bdd14"
Nov 01 20:02:28.783827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:28.783807    5066 scope.go:122] "RemoveContainer" containerID="0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2"
Nov 01 20:02:28.787101 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:28.784036    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:29.115255 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.115173    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hqhjj"]
Nov 01 20:02:29.118772 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:29.118734    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b66bc4_cb31_484b_b73f_80eaefeb3d78.slice/crio-464f4349660ba057dcd009386048027a7e7e23f3bd397f30b2520831f8619696 WatchSource:0}: Error finding container 464f4349660ba057dcd009386048027a7e7e23f3bd397f30b2520831f8619696: Status 404 returned error can't find the container with id 464f4349660ba057dcd009386048027a7e7e23f3bd397f30b2520831f8619696
Nov 01 20:02:29.120740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.120720    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:02:29.294747 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.294722    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fm8ml_067dc6af-f795-489f-820d-b81b63250a86/dns-node-resolver/0.log"
Nov 01 20:02:29.530783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.530750    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.530815    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.530870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.530884    5066 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.530904    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.530931    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:29.530988 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.530958    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.530939246 +0000 UTC m=+4.915836827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531001    5066 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531038    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531058    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d7b96b89b-7xj6x: secret "image-registry-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531068    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.531052782 +0000 UTC m=+4.915950364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531103    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls podName:42b9a858-43ed-4f9b-83b4-1a8da90ccdf3 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.531093924 +0000 UTC m=+4.915991496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls") pod "image-registry-5d7b96b89b-7xj6x" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3") : secret "image-registry-tls" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531039    5066 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 20:02:29.531215 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.531132    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.531126271 +0000 UTC m=+4.916023839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : configmap "kube-rbac-proxy" not found
Nov 01 20:02:29.533257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.533238    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2kwmf\" (UID: \"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:29.631826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.631799    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:29.631980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.631870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:29.631980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.631898    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:29.631980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.631915    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:29.631980 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.631938    5066 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632007    5066 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632018    5066 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632025    5066 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632037    5066 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd6f9db5-6m4r2: secret "image-registry-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632025    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls podName:c5fd4033-6a39-49fd-ad3d-f1bda1df12ad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.632006299 +0000 UTC m=+5.016903881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dbghn" (UID: "c5fd4033-6a39-49fd-ad3d-f1bda1df12ad") : secret "samples-operator-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632067    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.632058883 +0000 UTC m=+5.016956451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632076    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.632070881 +0000 UTC m=+5.016968449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:29.632132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:29.632086    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls podName:95635b7c-a032-45a4-9999-ef2a08af06e2 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:31.63208092 +0000 UTC m=+5.016978488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls") pod "image-registry-56cd6f9db5-6m4r2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2") : secret "image-registry-tls" not found
Nov 01 20:02:29.641399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.641378    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"
Nov 01 20:02:29.788627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.788558    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:02:29.789864 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:29.789834    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqhjj" event={"ID":"b7b66bc4-cb31-484b-b73f-80eaefeb3d78","Type":"ContainerStarted","Data":"464f4349660ba057dcd009386048027a7e7e23f3bd397f30b2520831f8619696"}
Nov 01 20:02:30.005551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:30.005516    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2kwmf"]
Nov 01 20:02:30.009471 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:30.009444    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b90dbb_0eac_4538_a594_7b1d3cbfc1ba.slice/crio-d2729904b928bedf2cf7cc98355c3e356f0e0af3260ba5f19bf8d5290eea963b WatchSource:0}: Error finding container d2729904b928bedf2cf7cc98355c3e356f0e0af3260ba5f19bf8d5290eea963b: Status 404 returned error can't find the container with id d2729904b928bedf2cf7cc98355c3e356f0e0af3260ba5f19bf8d5290eea963b
Nov 01 20:02:30.295199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:30.295170    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zq4jc_cb7319d0-ad0f-4af5-b0b6-31664beab34e/node-ca/0.log"
Nov 01 20:02:30.793661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:30.793639    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf" event={"ID":"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba","Type":"ContainerStarted","Data":"d2729904b928bedf2cf7cc98355c3e356f0e0af3260ba5f19bf8d5290eea963b"}
Nov 01 20:02:30.894619 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:30.894578    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8dbcd_c8a270bb-d842-48e6-a847-ceced3cc014a/serve-healthcheck-canary/0.log"
Nov 01 20:02:31.096623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.096551    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-996rs_3014c67d-90f7-41d9-8a27-1657568714cd/migrator/0.log"
Nov 01 20:02:31.295956 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.295922    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-c7hsh_35dd1a4d-103c-4343-a834-3479d75e59e0/migrator/0.log"
Nov 01 20:02:31.495467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.495435    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq_659b88a7-6607-43cf-a880-c3ee709c3862/kube-storage-version-migrator-operator/0.log"
Nov 01 20:02:31.550151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.550117    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:31.550309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.550159    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:31.550309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.550178    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:31.550309 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.550263    5066 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:31.550471 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.550378    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:31.550471 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.550427    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:35.550383586 +0000 UTC m=+8.935281162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:31.550627 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.550504    5066 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:31.550627 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.550594    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls podName:0a4a52df-9a7b-4332-92ec-8ddc8d299bad nodeName:}" failed. No retries permitted until 2026-11-01 20:02:35.55057899 +0000 UTC m=+8.935476571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5nd9r" (UID: "0a4a52df-9a7b-4332-92ec-8ddc8d299bad") : secret "cluster-monitoring-operator-tls" not found
Nov 01 20:02:31.550758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.550737    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42486c5c-cb87-4f5e-96b3-f957803d0b75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:31.552441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.552422    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"image-registry-5d7b96b89b-7xj6x\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") " pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:31.651271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.651232    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:31.651422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.651294    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:31.651422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.651320    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:31.651422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.651337    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:31.651535 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.651437    5066 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 20:02:31.651535 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.651463    5066 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 20:02:31.651535 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.651525    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls podName:1495ed93-b519-468e-8dd2-e40df2c3e979 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:35.651505187 +0000 UTC m=+9.036402769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls") pod "dns-default-94r67" (UID: "1495ed93-b519-468e-8dd2-e40df2c3e979") : secret "dns-default-metrics-tls" not found
Nov 01 20:02:31.651689 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.651547    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs podName:1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:35.651537344 +0000 UTC m=+9.036434915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs") pod "router-default-646fc8f47-znjf4" (UID: "1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6") : secret "router-metrics-certs-default" not found
Nov 01 20:02:31.653614 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.653591    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd4033-6a39-49fd-ad3d-f1bda1df12ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dbghn\" (UID: \"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:31.653679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.653651    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"image-registry-56cd6f9db5-6m4r2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") " pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:31.735441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.735409    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"
Nov 01 20:02:31.740674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.740641    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:31.744423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.744407    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:31.812279 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.812254    5066 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:31.812712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.812693    5066 scope.go:122] "RemoveContainer" containerID="0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2"
Nov 01 20:02:31.812925 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:31.812901    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:31.815302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.815274    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2kwmf" event={"ID":"d3b90dbb-0eac-4538-a594-7b1d3cbfc1ba","Type":"ContainerStarted","Data":"5fdf1994417c7f72457178fbbd2d8a978c69a367de02d2265afe659adb4557cf"}
Nov 01 20:02:31.826776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.826741    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqhjj" event={"ID":"b7b66bc4-cb31-484b-b73f-80eaefeb3d78","Type":"ContainerStarted","Data":"b03e876d5ec1d0fccb47c916283187e1d34bce6441adbad40241f5fba292ba2f"}
Nov 01 20:02:31.826850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:31.826816    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hqhjj" event={"ID":"b7b66bc4-cb31-484b-b73f-80eaefeb3d78","Type":"ContainerStarted","Data":"2629db2cc53b6a88b47474cfae3dceb327d7d8a9182034368a55802907ca4b11"}
Nov 01 20:02:32.117065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.116721    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn"]
Nov 01 20:02:32.170092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.170065    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:02:32.173313 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:32.173276    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95635b7c_a032_45a4_9999_ef2a08af06e2.slice/crio-a056936cb053e33e7c93bfce7985a66e45d4058428d9f487d05a66a7fee5e8cb WatchSource:0}: Error finding container a056936cb053e33e7c93bfce7985a66e45d4058428d9f487d05a66a7fee5e8cb: Status 404 returned error can't find the container with id a056936cb053e33e7c93bfce7985a66e45d4058428d9f487d05a66a7fee5e8cb
Nov 01 20:02:32.177610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.177590    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:02:32.181872 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:32.181848    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b9a858_43ed_4f9b_83b4_1a8da90ccdf3.slice/crio-c5d8ca2deb0bc07597c70fbb9ce27ec08ef96989fcaeb4c761a804f53cfd79e0 WatchSource:0}: Error finding container c5d8ca2deb0bc07597c70fbb9ce27ec08ef96989fcaeb4c761a804f53cfd79e0: Status 404 returned error can't find the container with id c5d8ca2deb0bc07597c70fbb9ce27ec08ef96989fcaeb4c761a804f53cfd79e0
Nov 01 20:02:32.831791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.831753    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2" event={"ID":"95635b7c-a032-45a4-9999-ef2a08af06e2","Type":"ContainerStarted","Data":"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"}
Nov 01 20:02:32.832254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.831816    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2" event={"ID":"95635b7c-a032-45a4-9999-ef2a08af06e2","Type":"ContainerStarted","Data":"a056936cb053e33e7c93bfce7985a66e45d4058428d9f487d05a66a7fee5e8cb"}
Nov 01 20:02:32.833254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.833226    5066 generic.go:356] "Generic (PLEG): container finished" podID="1946120d-acc5-423a-b8e8-eec897eb0bee" containerID="c397e07f0bf754b77cf9bb6dd2523a383e3310641c278c1161b79a2b0ee846c3" exitCode=0
Nov 01 20:02:32.833359 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.833300    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m49w5" event={"ID":"1946120d-acc5-423a-b8e8-eec897eb0bee","Type":"ContainerDied","Data":"c397e07f0bf754b77cf9bb6dd2523a383e3310641c278c1161b79a2b0ee846c3"}
Nov 01 20:02:32.833631 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.833614    5066 scope.go:122] "RemoveContainer" containerID="c397e07f0bf754b77cf9bb6dd2523a383e3310641c278c1161b79a2b0ee846c3"
Nov 01 20:02:32.834780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.834759    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" event={"ID":"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3","Type":"ContainerStarted","Data":"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"}
Nov 01 20:02:32.834897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.834803    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" event={"ID":"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3","Type":"ContainerStarted","Data":"c5d8ca2deb0bc07597c70fbb9ce27ec08ef96989fcaeb4c761a804f53cfd79e0"}
Nov 01 20:02:32.835886 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:32.835862    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn" event={"ID":"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad","Type":"ContainerStarted","Data":"e649adafc0644916a13bca77fd3142c51baba217d15f1694f843d0878a6ade92"}
Nov 01 20:02:33.821466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:33.821427    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:33.833671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:33.833645    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:33.834030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:33.834017    5066 scope.go:122] "RemoveContainer" containerID="0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2"
Nov 01 20:02:33.834226 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:33.834208    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dlrcw_openshift-console-operator(ce41e940-ffc1-4f08-93c8-3abd565d89e2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2"
Nov 01 20:02:33.841170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:33.841142    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m49w5" event={"ID":"1946120d-acc5-423a-b8e8-eec897eb0bee","Type":"ContainerStarted","Data":"c176ff4ab3c0d8e8cbb6bc8279574a02e65883375360306bff7de655f1dfa9c8"}
Nov 01 20:02:33.841301 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:33.841217    5066 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:02:34.073641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:34.073569    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktldr"
Nov 01 20:02:34.197961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:34.197931    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:35.127554 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.127529    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:35.324079 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.324053    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:35.324502 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.324306    5066 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:02:35.591200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.591125    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:35.591343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.591213    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:35.591343 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:35.591278    5066 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:35.591410 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:35.591351    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls podName:42486c5c-cb87-4f5e-96b3-f957803d0b75 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:43.591332406 +0000 UTC m=+16.976229975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sr5gb" (UID: "42486c5c-cb87-4f5e-96b3-f957803d0b75") : secret "insights-runtime-extractor-tls" not found
Nov 01 20:02:35.593549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.593523    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a4a52df-9a7b-4332-92ec-8ddc8d299bad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5nd9r\" (UID: \"0a4a52df-9a7b-4332-92ec-8ddc8d299bad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:35.636570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.636539    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"
Nov 01 20:02:35.692184 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.692153    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:35.692372 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.692208    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:35.694559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.694535    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1495ed93-b519-468e-8dd2-e40df2c3e979-metrics-tls\") pod \"dns-default-94r67\" (UID: \"1495ed93-b519-468e-8dd2-e40df2c3e979\") " pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:35.695327 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.694751    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6-metrics-certs\") pod \"router-default-646fc8f47-znjf4\" (UID: \"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6\") " pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:35.850035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.849954    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn" event={"ID":"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad","Type":"ContainerStarted","Data":"86415d95ae6ae4c2cfa8325e31e423bb249d6da483381adb9162019fe7d391ae"}
Nov 01 20:02:35.850035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.850008    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dbghn" event={"ID":"c5fd4033-6a39-49fd-ad3d-f1bda1df12ad","Type":"ContainerStarted","Data":"f0d32919d3d071f97177b00ff31f0a5a0025c30fac509add8084f6b37b158be5"}
Nov 01 20:02:35.934474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.934438    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:35.948756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:35.948725    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:36.020733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.020659    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r"]
Nov 01 20:02:36.326876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.326828    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-646fc8f47-znjf4"]
Nov 01 20:02:36.329576 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:36.329545    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e92c0a9_d5ef_49ff_905f_9dbdb07d5ad6.slice/crio-e784fd69b34b0c46046a1c9b89f7e78330fc7b079ff43dce3cdfed7418e6bd33 WatchSource:0}: Error finding container e784fd69b34b0c46046a1c9b89f7e78330fc7b079ff43dce3cdfed7418e6bd33: Status 404 returned error can't find the container with id e784fd69b34b0c46046a1c9b89f7e78330fc7b079ff43dce3cdfed7418e6bd33
Nov 01 20:02:36.377895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.377801    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94r67"]
Nov 01 20:02:36.380206 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:36.380181    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1495ed93_b519_468e_8dd2_e40df2c3e979.slice/crio-b9a49403981fc5d4e066d4b086862ffe21410355176c763754df52f0eb68ac9d WatchSource:0}: Error finding container b9a49403981fc5d4e066d4b086862ffe21410355176c763754df52f0eb68ac9d: Status 404 returned error can't find the container with id b9a49403981fc5d4e066d4b086862ffe21410355176c763754df52f0eb68ac9d
Nov 01 20:02:36.854664 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.854626    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94r67" event={"ID":"1495ed93-b519-468e-8dd2-e40df2c3e979","Type":"ContainerStarted","Data":"b9a49403981fc5d4e066d4b086862ffe21410355176c763754df52f0eb68ac9d"}
Nov 01 20:02:36.856059 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.856032    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-646fc8f47-znjf4" event={"ID":"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6","Type":"ContainerStarted","Data":"3a63779371cf62ca5f50edece978636874ae600f5919ba341793bb4c93d1c23e"}
Nov 01 20:02:36.856200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.856071    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-646fc8f47-znjf4" event={"ID":"1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6","Type":"ContainerStarted","Data":"e784fd69b34b0c46046a1c9b89f7e78330fc7b079ff43dce3cdfed7418e6bd33"}
Nov 01 20:02:36.857185 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.857157    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r" event={"ID":"0a4a52df-9a7b-4332-92ec-8ddc8d299bad","Type":"ContainerStarted","Data":"f5aa8c2083404d50d929c4cdcbe2c8e85ee9519e46c1d3b46b5fb503a4d0d4e9"}
Nov 01 20:02:36.935351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.935318    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:36.937899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:36.937873    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:37.406736 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:37.406704    5066 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-223.us-west-1.compute.internal" event="NodeReady"
Nov 01 20:02:38.254987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.254953    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh57k"
Nov 01 20:02:38.723800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.723771    5066 scope.go:122] "RemoveContainer" containerID="0f085d9845dda902be6aadfd4d0f95281487de7345c596bc96408f4059966ad2"
Nov 01 20:02:38.866861 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.866837    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:02:38.867039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.866932    5066 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:02:38.867039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.866961    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" event={"ID":"ce41e940-ffc1-4f08-93c8-3abd565d89e2","Type":"ContainerStarted","Data":"dc90811d42e1ccb4f583fd42618938447eeec364e2b89a71eeee79be4dc59d3a"}
Nov 01 20:02:38.867646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.867424    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:38.868315 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.868285    5066 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-dlrcw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Nov 01 20:02:38.868388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:38.868340    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw" podUID="ce41e940-ffc1-4f08-93c8-3abd565d89e2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Nov 01 20:02:39.873006 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:39.872966    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94r67" event={"ID":"1495ed93-b519-468e-8dd2-e40df2c3e979","Type":"ContainerStarted","Data":"2411451f04f74614f59d2ebc30f61a01bcf0e1dedf25895d1b231679ada69a76"}
Nov 01 20:02:39.873428 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:39.873029    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94r67" event={"ID":"1495ed93-b519-468e-8dd2-e40df2c3e979","Type":"ContainerStarted","Data":"bba93e717f6e9c3fd9deceaab635922c9c05733e4b891b0785aa8559f8899e58"}
Nov 01 20:02:39.873428 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:39.873060    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:39.874601 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:39.874580    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5nd9r" event={"ID":"0a4a52df-9a7b-4332-92ec-8ddc8d299bad","Type":"ContainerStarted","Data":"20ea5623501f6144c5c6562811ec9357d8e77f48a12cfae3abdbf1d2e5a88f98"}
Nov 01 20:02:39.879036 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:39.879016    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dlrcw"
Nov 01 20:02:40.943273 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:40.943246    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:40.943775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:40.943353    5066 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 20:02:40.944585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:40.944563    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-646fc8f47-znjf4"
Nov 01 20:02:43.665633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:43.665599    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:43.667817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:43.667788    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42486c5c-cb87-4f5e-96b3-f957803d0b75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sr5gb\" (UID: \"42486c5c-cb87-4f5e-96b3-f957803d0b75\") " pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:43.748965 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:43.748934    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sr5gb"
Nov 01 20:02:44.144602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.144566    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sr5gb"]
Nov 01 20:02:44.149840 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:44.149811    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42486c5c_cb87_4f5e_96b3_f957803d0b75.slice/crio-5fef1996281b1e6f7e53243e5c54de224e67271ca36b6900628287410016a11b WatchSource:0}: Error finding container 5fef1996281b1e6f7e53243e5c54de224e67271ca36b6900628287410016a11b: Status 404 returned error can't find the container with id 5fef1996281b1e6f7e53243e5c54de224e67271ca36b6900628287410016a11b
Nov 01 20:02:44.896417 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.896380    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr5gb" event={"ID":"42486c5c-cb87-4f5e-96b3-f957803d0b75","Type":"ContainerStarted","Data":"7020f0c3278770af42f4dbf93fa95abf2763d16329ecffc216a50b20325a9283"}
Nov 01 20:02:44.896827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.896454    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr5gb" event={"ID":"42486c5c-cb87-4f5e-96b3-f957803d0b75","Type":"ContainerStarted","Data":"5fef1996281b1e6f7e53243e5c54de224e67271ca36b6900628287410016a11b"}
Nov 01 20:02:44.921605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.921578    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"]
Nov 01 20:02:44.947416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.947393    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:44.971594 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.971571    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-jwwn7"]
Nov 01 20:02:44.994115 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.994093    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:02:44.994253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:44.994232    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.019904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.019882    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 20:02:45.020022 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.019924    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kcmjb\""
Nov 01 20:02:45.040235 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.040038    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ff88b4ff9-w88cd"]
Nov 01 20:02:45.048411 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.048393    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 20:02:45.050266 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.050248    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 20:02:45.059716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.059695    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.068487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.068471    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-96lsv\""
Nov 01 20:02:45.069373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.069357    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"]
Nov 01 20:02:45.076289 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.076270    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/689090e6-11bf-4c1d-8d64-9d38d2ab888a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lg5b7\" (UID: \"689090e6-11bf-4c1d-8d64-9d38d2ab888a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:45.076365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.076305    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a365b94b-41e4-4df7-b58d-88feff980742-tmp\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.076365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.076325    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tjk\" (UniqueName: \"kubernetes.io/projected/a365b94b-41e4-4df7-b58d-88feff980742-kube-api-access-f4tjk\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.092152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.092128    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-jwwn7"]
Nov 01 20:02:45.153110 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.153039    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff88b4ff9-w88cd"]
Nov 01 20:02:45.176826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.176803    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-registry-certificates\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.176933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.176831    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-installation-pull-secrets\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.176933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.176849    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9frv\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-kube-api-access-d9frv\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177004    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/689090e6-11bf-4c1d-8d64-9d38d2ab888a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lg5b7\" (UID: \"689090e6-11bf-4c1d-8d64-9d38d2ab888a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:45.177078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177030    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-bound-sa-token\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177080    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a365b94b-41e4-4df7-b58d-88feff980742-tmp\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.177168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177110    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tjk\" (UniqueName: \"kubernetes.io/projected/a365b94b-41e4-4df7-b58d-88feff980742-kube-api-access-f4tjk\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.177168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177151    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-image-registry-private-configuration\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177169    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bdbb26b-b5da-4933-9689-890018f14188-ca-trust-extracted\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177189    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-registry-tls\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177303    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-trusted-ca\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.177626 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.177607    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a365b94b-41e4-4df7-b58d-88feff980742-tmp\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.179472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.179453    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/689090e6-11bf-4c1d-8d64-9d38d2ab888a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lg5b7\" (UID: \"689090e6-11bf-4c1d-8d64-9d38d2ab888a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:45.233176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.233150    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tjk\" (UniqueName: \"kubernetes.io/projected/a365b94b-41e4-4df7-b58d-88feff980742-kube-api-access-f4tjk\") pod \"downloads-6dd789c8b7-jwwn7\" (UID: \"a365b94b-41e4-4df7-b58d-88feff980742\") " pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.270045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.270025    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:45.278088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278061    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-trusted-ca\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278109    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-registry-certificates\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278128    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-installation-pull-secrets\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278150    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9frv\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-kube-api-access-d9frv\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278182    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-bound-sa-token\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278262    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-image-registry-private-configuration\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278312    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bdbb26b-b5da-4933-9689-890018f14188-ca-trust-extracted\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278343    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-registry-tls\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.278881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.278853    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bdbb26b-b5da-4933-9689-890018f14188-ca-trust-extracted\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.279412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.279386    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-registry-certificates\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.279606 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.279584    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbb26b-b5da-4933-9689-890018f14188-trusted-ca\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.280554 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.280537    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-installation-pull-secrets\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.280792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.280769    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bdbb26b-b5da-4933-9689-890018f14188-image-registry-private-configuration\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.281369 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.281352    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-registry-tls\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.320509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.320472    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:02:45.338543 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.338516    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-bound-sa-token\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.339007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.338985    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9frv\" (UniqueName: \"kubernetes.io/projected/2bdbb26b-b5da-4933-9689-890018f14188-kube-api-access-d9frv\") pod \"image-registry-7ff88b4ff9-w88cd\" (UID: \"2bdbb26b-b5da-4933-9689-890018f14188\") " pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.384504 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.384472    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.653060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.653031    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"]
Nov 01 20:02:45.706418 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.706360    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-jwwn7"]
Nov 01 20:02:45.710331 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:45.710304    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689090e6_11bf_4c1d_8d64_9d38d2ab888a.slice/crio-2f22c52ea5d6a096de711c13f17416354c68807c235958eaedb6c77c95f28edf WatchSource:0}: Error finding container 2f22c52ea5d6a096de711c13f17416354c68807c235958eaedb6c77c95f28edf: Status 404 returned error can't find the container with id 2f22c52ea5d6a096de711c13f17416354c68807c235958eaedb6c77c95f28edf
Nov 01 20:02:45.711075 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:45.710975    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda365b94b_41e4_4df7_b58d_88feff980742.slice/crio-b77f5043c2adafaf949556674b340597071536a1bf477e6fbb7d0edcf1ff98bf WatchSource:0}: Error finding container b77f5043c2adafaf949556674b340597071536a1bf477e6fbb7d0edcf1ff98bf: Status 404 returned error can't find the container with id b77f5043c2adafaf949556674b340597071536a1bf477e6fbb7d0edcf1ff98bf
Nov 01 20:02:45.789278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.789253    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff88b4ff9-w88cd"]
Nov 01 20:02:45.791759 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:45.791738    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdbb26b_b5da_4933_9689_890018f14188.slice/crio-ede344bb311b7c4fd591e83aab3573551efdaa43deccc3b2c958134025f47f8d WatchSource:0}: Error finding container ede344bb311b7c4fd591e83aab3573551efdaa43deccc3b2c958134025f47f8d: Status 404 returned error can't find the container with id ede344bb311b7c4fd591e83aab3573551efdaa43deccc3b2c958134025f47f8d
Nov 01 20:02:45.901852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.901824    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-jwwn7" event={"ID":"a365b94b-41e4-4df7-b58d-88feff980742","Type":"ContainerStarted","Data":"b77f5043c2adafaf949556674b340597071536a1bf477e6fbb7d0edcf1ff98bf"}
Nov 01 20:02:45.903436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.903408    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd" event={"ID":"2bdbb26b-b5da-4933-9689-890018f14188","Type":"ContainerStarted","Data":"4296e054aa7041e1b11b30ea1dc85e38e470c6e754befc2aa47bc083842d74a1"}
Nov 01 20:02:45.903542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.903459    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd" event={"ID":"2bdbb26b-b5da-4933-9689-890018f14188","Type":"ContainerStarted","Data":"ede344bb311b7c4fd591e83aab3573551efdaa43deccc3b2c958134025f47f8d"}
Nov 01 20:02:45.903542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.903488    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:02:45.904542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:45.904518    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7" event={"ID":"689090e6-11bf-4c1d-8d64-9d38d2ab888a","Type":"ContainerStarted","Data":"2f22c52ea5d6a096de711c13f17416354c68807c235958eaedb6c77c95f28edf"}
Nov 01 20:02:46.018985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:46.018921    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd" podStartSLOduration=2.018909478 podStartE2EDuration="2.018909478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:46.018541936 +0000 UTC m=+19.403439527" watchObservedRunningTime="2026-11-01 20:02:46.018909478 +0000 UTC m=+19.403807062"
Nov 01 20:02:46.911665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:46.911628    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr5gb" event={"ID":"42486c5c-cb87-4f5e-96b3-f957803d0b75","Type":"ContainerStarted","Data":"7b05701ea80adb6e9bd978e3a34986a6d47fba91551fcb251024a4de994b909e"}
Nov 01 20:02:47.919298 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:47.919269    5066 patch_prober.go:29] interesting pod/image-registry-5d7b96b89b-7xj6x container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 20:02:47.919737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:47.919320    5066 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:02:48.922536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:48.922481    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sr5gb" event={"ID":"42486c5c-cb87-4f5e-96b3-f957803d0b75","Type":"ContainerStarted","Data":"28db6465f4b748de165344983f65474304ee066b57f9da9c4d40c409693755f7"}
Nov 01 20:02:48.924022 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:48.923999    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7" event={"ID":"689090e6-11bf-4c1d-8d64-9d38d2ab888a","Type":"ContainerStarted","Data":"89410e6cb1773273a2d3cb2628964b1bade070b30e92d97e09350f7a7c21d5d9"}
Nov 01 20:02:48.924137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:48.924098    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:48.928472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:48.928453    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7"
Nov 01 20:02:49.151634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:49.151585    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lg5b7" podStartSLOduration=2.933977236 podStartE2EDuration="5.151570219s" totalImagesPullingTime="2.217592983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:49.151157194 +0000 UTC m=+22.536054794" watchObservedRunningTime="2026-11-01 20:02:49.151570219 +0000 UTC m=+22.536467809"
Nov 01 20:02:49.880334 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:49.880304    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94r67"
Nov 01 20:02:51.628760 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:02:51.628726    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:02:54.849348 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:54.849318    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:02:55.000658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:55.000621    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:02:55.130599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:55.130522    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q4r47"
Nov 01 20:02:56.996735 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:56.996704    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s2bkv"]
Nov 01 20:02:56.999982 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:56.999957    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.067286 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.067252    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 20:02:57.102188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.102158    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 20:02:57.102188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.102164    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\""
Nov 01 20:02:57.151921 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.151896    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 20:02:57.152086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.151990    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 20:02:57.183673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183641    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-wtmp\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.183815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183677    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.183815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183707    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-sys\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.183938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183839    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-root\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.183938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183873    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-textfile\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.184051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.183966    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.184051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.184020    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-tls\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.184142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.184049    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hrl\" (UniqueName: \"kubernetes.io/projected/642b34b2-e573-4b35-b98f-24c98279ccf5-kube-api-access-p6hrl\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.184142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.184085    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-metrics-client-ca\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284596    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-tls\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284647    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hrl\" (UniqueName: \"kubernetes.io/projected/642b34b2-e573-4b35-b98f-24c98279ccf5-kube-api-access-p6hrl\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284665    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-metrics-client-ca\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284720    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-wtmp\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284781    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284840    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-sys\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284884    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-root\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.284924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.284910    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-textfile\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285033    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285037    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-sys\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285103    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-wtmp\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285212    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/642b34b2-e573-4b35-b98f-24c98279ccf5-root\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285405 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285377    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-textfile\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285482    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-metrics-client-ca\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.285719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.285666    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.287229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.287209    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.287378 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.287362    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/642b34b2-e573-4b35-b98f-24c98279ccf5-node-exporter-tls\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.344242 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.344213    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hrl\" (UniqueName: \"kubernetes.io/projected/642b34b2-e573-4b35-b98f-24c98279ccf5-kube-api-access-p6hrl\") pod \"node-exporter-s2bkv\" (UID: \"642b34b2-e573-4b35-b98f-24c98279ccf5\") " pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.614767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.614688    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2bkv"
Nov 01 20:02:57.627640 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:02:57.627609    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642b34b2_e573_4b35_b98f_24c98279ccf5.slice/crio-9ff2ef4f5db8007b1808cd3ef52eb16df3cc0729167ae6cbe86df4f5154850f1 WatchSource:0}: Error finding container 9ff2ef4f5db8007b1808cd3ef52eb16df3cc0729167ae6cbe86df4f5154850f1: Status 404 returned error can't find the container with id 9ff2ef4f5db8007b1808cd3ef52eb16df3cc0729167ae6cbe86df4f5154850f1
Nov 01 20:02:57.966271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:57.966242    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2bkv" event={"ID":"642b34b2-e573-4b35-b98f-24c98279ccf5","Type":"ContainerStarted","Data":"9ff2ef4f5db8007b1808cd3ef52eb16df3cc0729167ae6cbe86df4f5154850f1"}
Nov 01 20:02:59.976788 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:59.976750    5066 generic.go:356] "Generic (PLEG): container finished" podID="642b34b2-e573-4b35-b98f-24c98279ccf5" containerID="2bd91ebcf09d33f1af33d750913bbd3fb36274f7ef40ea989b4e6882e1abc163" exitCode=0
Nov 01 20:02:59.977265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:02:59.976830    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2bkv" event={"ID":"642b34b2-e573-4b35-b98f-24c98279ccf5","Type":"ContainerDied","Data":"2bd91ebcf09d33f1af33d750913bbd3fb36274f7ef40ea989b4e6882e1abc163"}
Nov 01 20:03:00.984119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:00.984075    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2bkv" event={"ID":"642b34b2-e573-4b35-b98f-24c98279ccf5","Type":"ContainerStarted","Data":"e78893ba9fb400e995277355f987154f6dc39da60b4810e60ef07efeefa33d07"}
Nov 01 20:03:00.984565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:00.984148    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2bkv" event={"ID":"642b34b2-e573-4b35-b98f-24c98279ccf5","Type":"ContainerStarted","Data":"b3ebc178608839e169f2a9fe40d55a4dd72fca9b15fb5e369cb92a74719e6874"}
Nov 01 20:03:01.149570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:01.149506    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s2bkv" podStartSLOduration=3.6281880109999998 podStartE2EDuration="5.1494816s" totalImagesPullingTime="1.445710497s" totalInitContainerRuntime="75.583092ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:01.147810111 +0000 UTC m=+34.532707701" watchObservedRunningTime="2026-11-01 20:03:01.1494816 +0000 UTC m=+34.534379185"
Nov 01 20:03:05.004949 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:05.004870    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-jwwn7" event={"ID":"a365b94b-41e4-4df7-b58d-88feff980742","Type":"ContainerStarted","Data":"7c36e032024e9c0bc5b97d35c1fc09f326f1a8160888ad58164f2d8e1c5eba29"}
Nov 01 20:03:05.005441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:05.005417    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:03:05.027339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:05.027316    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-jwwn7"
Nov 01 20:03:05.219173 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:05.219118    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-jwwn7" podStartSLOduration=2.201131409 podStartE2EDuration="21.219099066s" totalImagesPullingTime="19.017967657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:05.16805686 +0000 UTC m=+38.552954450" watchObservedRunningTime="2026-11-01 20:03:05.219099066 +0000 UTC m=+38.603996655"
Nov 01 20:03:06.916139 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:06.916110    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7ff88b4ff9-w88cd"
Nov 01 20:03:07.133459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:07.133421    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:03:10.016411 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.016294    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2" podUID="95635b7c-a032-45a4-9999-ef2a08af06e2" containerName="registry" containerID="cri-o://f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6" gracePeriod=30
Nov 01 20:03:10.319756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.319728    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:03:10.405735 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.405701    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.405927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.405755    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.405927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.405802    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.405927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.405843    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.405927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.405871    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.406357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.406335    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.406887 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.406827    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:10.407684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.407031    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.407684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.407139    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") pod \"95635b7c-a032-45a4-9999-ef2a08af06e2\" (UID: \"95635b7c-a032-45a4-9999-ef2a08af06e2\") "
Nov 01 20:03:10.407684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.407393    5066 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-trusted-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.408031 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.407893    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:10.411858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.411828    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:10.412107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.412079    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:10.412196 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.412084    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:10.412859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.412827    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:10.414319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.414292    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:03:10.417953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.417929    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf" pod "95635b7c-a032-45a4-9999-ef2a08af06e2" (UID: "95635b7c-a032-45a4-9999-ef2a08af06e2"). InnerVolumeSpecName "kube-api-access-59pbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:10.508786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508747    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pbf\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-kube-api-access-59pbf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.508786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508778    5066 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-installation-pull-secrets\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.508786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508791    5066 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95635b7c-a032-45a4-9999-ef2a08af06e2-ca-trust-extracted\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.509103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508804    5066 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-bound-sa-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.509103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508813    5066 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95635b7c-a032-45a4-9999-ef2a08af06e2-image-registry-private-configuration\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.509103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508822    5066 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:10.509103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:10.508844    5066 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95635b7c-a032-45a4-9999-ef2a08af06e2-registry-certificates\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:11.034427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.034355    5066 generic.go:356] "Generic (PLEG): container finished" podID="95635b7c-a032-45a4-9999-ef2a08af06e2" containerID="f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6" exitCode=0
Nov 01 20:03:11.034966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.034424    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2"
Nov 01 20:03:11.034966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.034435    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2" event={"ID":"95635b7c-a032-45a4-9999-ef2a08af06e2","Type":"ContainerDied","Data":"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"}
Nov 01 20:03:11.034966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.034513    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd6f9db5-6m4r2" event={"ID":"95635b7c-a032-45a4-9999-ef2a08af06e2","Type":"ContainerDied","Data":"a056936cb053e33e7c93bfce7985a66e45d4058428d9f487d05a66a7fee5e8cb"}
Nov 01 20:03:11.034966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.034522    5066 scope.go:122] "RemoveContainer" containerID="f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"
Nov 01 20:03:11.049892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.049865    5066 scope.go:122] "RemoveContainer" containerID="f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"
Nov 01 20:03:11.050242 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:03:11.050212    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6\": container with ID starting with f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6 not found: ID does not exist" containerID="f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"
Nov 01 20:03:11.050338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.050251    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6"} err="failed to get container status \"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6\": rpc error: code = NotFound desc = could not find container \"f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6\": container with ID starting with f373cda9e2b12860cb7ec058bf6fa9a3d14de43fe649cbf9038e73acf9f78aa6 not found: ID does not exist"
Nov 01 20:03:11.247899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.247863    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:03:11.297526 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:11.297432    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56cd6f9db5-6m4r2"]
Nov 01 20:03:12.730291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:12.730250    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95635b7c-a032-45a4-9999-ef2a08af06e2" path="/var/lib/kubelet/pods/95635b7c-a032-45a4-9999-ef2a08af06e2/volumes"
Nov 01 20:03:13.242697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:13.242664    5066 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktldr_8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:13.484457 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:13.484427    5066 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktldr_8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/konnectivity-agent/0.log" limit=1024
Nov 01 20:03:17.440888 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.440859    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:03:17.441359 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.441084    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" podUID="c34b1735-2c5a-442b-a067-2c165eb00003" containerName="insights-gathering" containerID="cri-o://ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4" gracePeriod=30
Nov 01 20:03:17.745705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.745678    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:03:17.873389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873358    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") pod \"c34b1735-2c5a-442b-a067-2c165eb00003\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") "
Nov 01 20:03:17.873566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873400    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") pod \"c34b1735-2c5a-442b-a067-2c165eb00003\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") "
Nov 01 20:03:17.873566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873467    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") pod \"c34b1735-2c5a-442b-a067-2c165eb00003\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") "
Nov 01 20:03:17.873566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873516    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") pod \"c34b1735-2c5a-442b-a067-2c165eb00003\" (UID: \"c34b1735-2c5a-442b-a067-2c165eb00003\") "
Nov 01 20:03:17.873797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873764    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle" pod "c34b1735-2c5a-442b-a067-2c165eb00003" (UID: "c34b1735-2c5a-442b-a067-2c165eb00003"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:17.873914 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873780    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path" pod "c34b1735-2c5a-442b-a067-2c165eb00003" (UID: "c34b1735-2c5a-442b-a067-2c165eb00003"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:03:17.873972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.873922    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle" pod "c34b1735-2c5a-442b-a067-2c165eb00003" (UID: "c34b1735-2c5a-442b-a067-2c165eb00003"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:17.875826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.875791    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56" pod "c34b1735-2c5a-442b-a067-2c165eb00003" (UID: "c34b1735-2c5a-442b-a067-2c165eb00003"). InnerVolumeSpecName "kube-api-access-wnn56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:17.974423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.974398    5066 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-trusted-ca-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:17.974423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.974424    5066 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c34b1735-2c5a-442b-a067-2c165eb00003-archives-path\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:17.974596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.974434    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnn56\" (UniqueName: \"kubernetes.io/projected/c34b1735-2c5a-442b-a067-2c165eb00003-kube-api-access-wnn56\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:17.974596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:17.974444    5066 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c34b1735-2c5a-442b-a067-2c165eb00003-service-ca-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:18.066396 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.066355    5066 generic.go:356] "Generic (PLEG): container finished" podID="c34b1735-2c5a-442b-a067-2c165eb00003" containerID="ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4" exitCode=2
Nov 01 20:03:18.066569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.066402    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw"
Nov 01 20:03:18.066569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.066414    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" event={"ID":"c34b1735-2c5a-442b-a067-2c165eb00003","Type":"ContainerDied","Data":"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"}
Nov 01 20:03:18.066569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.066459    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4d7xr-lr6rw" event={"ID":"c34b1735-2c5a-442b-a067-2c165eb00003","Type":"ContainerDied","Data":"4710e4db1d54b07afa47261de8898fea8e17d9168dbcc30a2c803355c36abfd7"}
Nov 01 20:03:18.066569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.066477    5066 scope.go:122] "RemoveContainer" containerID="ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"
Nov 01 20:03:18.074717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.074699    5066 scope.go:122] "RemoveContainer" containerID="ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"
Nov 01 20:03:18.074969 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:03:18.074950    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4\": container with ID starting with ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4 not found: ID does not exist" containerID="ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"
Nov 01 20:03:18.075039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.074979    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4"} err="failed to get container status \"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4\": rpc error: code = NotFound desc = could not find container \"ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4\": container with ID starting with ab340324810bda5600f28132b1483a35b78f288af103756e7bf161eda23aaae4 not found: ID does not exist"
Nov 01 20:03:18.227842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.227780    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:03:18.277475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.277449    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4d7xr-lr6rw"]
Nov 01 20:03:18.728765 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:18.728736    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34b1735-2c5a-442b-a067-2c165eb00003" path="/var/lib/kubelet/pods/c34b1735-2c5a-442b-a067-2c165eb00003/volumes"
Nov 01 20:03:21.627315 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:03:21.627283    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:03:32.157003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.156962    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerName="registry" containerID="cri-o://c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0" gracePeriod=30
Nov 01 20:03:32.423077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.423013    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:03:32.592933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.592899    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.592941    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.592973    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593011    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593056    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593137    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593199    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593606    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") pod \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\" (UID: \"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3\") "
Nov 01 20:03:32.593773 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593659    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:32.593902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.593881    5066 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-trusted-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.594265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.594224    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:03:32.595580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.595556    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:32.595580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.595573    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:32.595774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.595753    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:32.595774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.595763    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:32.596012 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.595993    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "kube-api-access-dhc4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:32.599989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.599961    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted" pod "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" (UID: "42b9a858-43ed-4f9b-83b4-1a8da90ccdf3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.694938    5066 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-image-registry-private-configuration\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.694971    5066 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-certificates\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.694983    5066 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-installation-pull-secrets\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.694994    5066 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-registry-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.695003    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc4c\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-kube-api-access-dhc4c\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.695011    5066 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-bound-sa-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:32.695014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:32.695020    5066 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3-ca-trust-extracted\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:03:33.126313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.126281    5066 generic.go:356] "Generic (PLEG): container finished" podID="33e840a4-f321-43d0-9864-5321d68fd28e" containerID="2fbcf5e0b5c75dd51478eaf580ced053e27dfb06c37aa5f728d299facf3715b6" exitCode=0
Nov 01 20:03:33.126517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.126358    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w" event={"ID":"33e840a4-f321-43d0-9864-5321d68fd28e","Type":"ContainerDied","Data":"2fbcf5e0b5c75dd51478eaf580ced053e27dfb06c37aa5f728d299facf3715b6"}
Nov 01 20:03:33.126727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.126707    5066 scope.go:122] "RemoveContainer" containerID="2fbcf5e0b5c75dd51478eaf580ced053e27dfb06c37aa5f728d299facf3715b6"
Nov 01 20:03:33.127662 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.127639    5066 generic.go:356] "Generic (PLEG): container finished" podID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerID="c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0" exitCode=0
Nov 01 20:03:33.127772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.127686    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" event={"ID":"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3","Type":"ContainerDied","Data":"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"}
Nov 01 20:03:33.127772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.127701    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x"
Nov 01 20:03:33.127772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.127718    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7b96b89b-7xj6x" event={"ID":"42b9a858-43ed-4f9b-83b4-1a8da90ccdf3","Type":"ContainerDied","Data":"c5d8ca2deb0bc07597c70fbb9ce27ec08ef96989fcaeb4c761a804f53cfd79e0"}
Nov 01 20:03:33.127772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.127738    5066 scope.go:122] "RemoveContainer" containerID="c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"
Nov 01 20:03:33.135567 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.135550    5066 scope.go:122] "RemoveContainer" containerID="c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"
Nov 01 20:03:33.135796 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:03:33.135775    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0\": container with ID starting with c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0 not found: ID does not exist" containerID="c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"
Nov 01 20:03:33.135859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.135807    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0"} err="failed to get container status \"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0\": rpc error: code = NotFound desc = could not find container \"c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0\": container with ID starting with c29b8e2bc682351739542d8fee1baaf3be7e365c097e4b09105135fb5e676ac0 not found: ID does not exist"
Nov 01 20:03:33.351523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.351478    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:03:33.377446 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:33.377384    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d7b96b89b-7xj6x"]
Nov 01 20:03:34.134569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:34.134537    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2f66w" event={"ID":"33e840a4-f321-43d0-9864-5321d68fd28e","Type":"ContainerStarted","Data":"c7cb724eae782de368a48c8455a973c2af22e53a30b50689dfb8817ad399c91b"}
Nov 01 20:03:34.729229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:34.729197    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" path="/var/lib/kubelet/pods/42b9a858-43ed-4f9b-83b4-1a8da90ccdf3/volumes"
Nov 01 20:03:38.150916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:38.150882    5066 generic.go:356] "Generic (PLEG): container finished" podID="659b88a7-6607-43cf-a880-c3ee709c3862" containerID="4738e02a9c8bb2ee92737319a41249ee5f8f7d8388505be1b5524188fde211d5" exitCode=0
Nov 01 20:03:38.151361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:38.150953    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq" event={"ID":"659b88a7-6607-43cf-a880-c3ee709c3862","Type":"ContainerDied","Data":"4738e02a9c8bb2ee92737319a41249ee5f8f7d8388505be1b5524188fde211d5"}
Nov 01 20:03:38.151361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:38.151292    5066 scope.go:122] "RemoveContainer" containerID="4738e02a9c8bb2ee92737319a41249ee5f8f7d8388505be1b5524188fde211d5"
Nov 01 20:03:39.156511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:03:39.156468    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g2nnq" event={"ID":"659b88a7-6607-43cf-a880-c3ee709c3862","Type":"ContainerStarted","Data":"938e851a0b6bc63d7a8be226105ee3f417c945cee7ca12783de5d50a57e159b3"}
Nov 01 20:03:51.627229 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:03:51.627194    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:04:21.626948 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:04:21.626913    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:04:51.626696 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:04:51.626619    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:05:21.627338 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:05:21.627300    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:05:51.629333 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:05:51.627609    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:05:57.242222 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242188    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"]
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242511    5066 cpu_manager.go:414] "containerMap: removing container" podUID="95635b7c-a032-45a4-9999-ef2a08af06e2" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242523    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95635b7c-a032-45a4-9999-ef2a08af06e2" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242542    5066 cpu_manager.go:414] "containerMap: removing container" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242546    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242560    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c34b1735-2c5a-442b-a067-2c165eb00003" containerName="insights-gathering"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242566    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34b1735-2c5a-442b-a067-2c165eb00003" containerName="insights-gathering"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242624    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="95635b7c-a032-45a4-9999-ef2a08af06e2" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242635    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b9a858-43ed-4f9b-83b4-1a8da90ccdf3" containerName="registry"
Nov 01 20:05:57.242716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.242642    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34b1735-2c5a-442b-a067-2c165eb00003" containerName="insights-gathering"
Nov 01 20:05:57.244746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.244728    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.296377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.296354    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 20:05:57.296377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.296361    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 20:05:57.329973 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.329947    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-qhjhf\""
Nov 01 20:05:57.347357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.347327    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"]
Nov 01 20:05:57.348265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.348244    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"]
Nov 01 20:05:57.350413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.350396    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.357432 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.357410    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.357528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.357447    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.357528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.357477    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.394712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.394693    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"]
Nov 01 20:05:57.402115 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.402098    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-vndjh\""
Nov 01 20:05:57.435349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.435325    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"]
Nov 01 20:05:57.437451 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.437437    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.457836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.457816    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.457933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.457860    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.457933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.457895    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.458039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.457939    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrz6\" (UniqueName: \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.458039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.458003    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.458140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.458082    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.458140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.458125    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.458271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.458251    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.495364 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.495312    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"]
Nov 01 20:05:57.502535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.502517    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2kfch\""
Nov 01 20:05:57.514121 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.514099    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.556162 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.556141    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:05:57.558988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.558969    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.559043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559020    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrz6\" (UniqueName: \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.559081 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559045    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.559116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559088    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.559154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559120    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.559154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559143    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlmn\" (UniqueName: \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.559318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559302    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.559519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.559480    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.617771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.617746    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrz6\" (UniqueName: \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.659830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.659805    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.659970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.659847    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.659970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.659871    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlmn\" (UniqueName: \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.660163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.660141    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.660226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.660188    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.662757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.662730    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:05:57.717837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.717812    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlmn\" (UniqueName: \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.750099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.750047    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:05:57.923608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:57.923587    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"]
Nov 01 20:05:57.925537 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:05:57.925460    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43822ed_2c51_4bf7_999d_937564029b5a.slice/crio-5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302 WatchSource:0}: Error finding container 5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302: Status 404 returned error can't find the container with id 5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302
Nov 01 20:05:58.024940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.024885    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"]
Nov 01 20:05:58.027455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:05:58.027432    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867fc386_24da_497e_91ab_dd6ca920895b.slice/crio-fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b WatchSource:0}: Error finding container fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b: Status 404 returned error can't find the container with id fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b
Nov 01 20:05:58.110797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.110777    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"]
Nov 01 20:05:58.113925 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:05:58.113898    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045a4822_7409_4b83_8341_70f3005bc199.slice/crio-3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e WatchSource:0}: Error finding container 3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e: Status 404 returned error can't find the container with id 3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e
Nov 01 20:05:58.441544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.441466    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"]
Nov 01 20:05:58.444509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.444475    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.496575 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.496551    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kjxbj\""
Nov 01 20:05:58.502537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.502515    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"]
Nov 01 20:05:58.568145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.568121    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.568232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.568152    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7l7\" (UniqueName: \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.568232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.568180    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.669572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.669549    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.669666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.669578    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7l7\" (UniqueName: \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.669666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.669602    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.669915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.669899    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.669958 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.669941    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.693067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.693011    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerStarted","Data":"fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b"}
Nov 01 20:05:58.694060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.694039    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg" event={"ID":"d43822ed-2c51-4bf7-999d-937564029b5a","Type":"ContainerStarted","Data":"5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302"}
Nov 01 20:05:58.695109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.695090    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerStarted","Data":"3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e"}
Nov 01 20:05:58.726390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.726365    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7l7\" (UniqueName: \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:58.757194 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:58.757167    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:05:59.121334 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.121295    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"]
Nov 01 20:05:59.134565 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:05:59.134531    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca761b7_6e22_4c33_a5a1_0c30e2dcd9d1.slice/crio-ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779 WatchSource:0}: Error finding container ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779: Status 404 returned error can't find the container with id ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779
Nov 01 20:05:59.705155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.705122    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerStarted","Data":"ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779"}
Nov 01 20:05:59.750705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.750677    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"]
Nov 01 20:05:59.753408 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.753386    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.854819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.854765    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-k6stf\""
Nov 01 20:05:59.856704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.856674    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"]
Nov 01 20:05:59.879850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.879803    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.880027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.879863    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dr4n\" (UniqueName: \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.880027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.879927    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.980597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.980562    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dr4n\" (UniqueName: \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.980782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.980637    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.980782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.980732    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.981161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.981088    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:05:59.981161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:05:59.981128    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:06:00.052349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:00.052318    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dr4n\" (UniqueName: \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:06:00.065792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:00.065753    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:06:00.441512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:00.441416    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"]
Nov 01 20:06:00.477564 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:00.477532    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06685dbf_0ee3_4ace_b8cf_8cf4afa98cc8.slice/crio-d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984 WatchSource:0}: Error finding container d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984: Status 404 returned error can't find the container with id d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984
Nov 01 20:06:00.712990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:00.712907    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerStarted","Data":"d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984"}
Nov 01 20:06:02.317242 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.317207    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"]
Nov 01 20:06:02.320682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.320115    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.403802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.403771    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.403978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.403847    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.403978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.403960    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6kq\" (UniqueName: \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.422643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.422614    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-v42cj\""
Nov 01 20:06:02.426004 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.425619    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"]
Nov 01 20:06:02.504678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.504645    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.504817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.504722    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.504885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.504822    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6kq\" (UniqueName: \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.505438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.505317    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.505438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.505364    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.566029 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.565939    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6kq\" (UniqueName: \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:02.637795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:02.637526    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:03.030198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:03.030086    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"]
Nov 01 20:06:03.042477 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:03.042444    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb16331f_29b9_43e0_a9b8_0d398d85ffb1.slice/crio-e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159 WatchSource:0}: Error finding container e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159: Status 404 returned error can't find the container with id e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159
Nov 01 20:06:03.742106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:03.742033    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerStarted","Data":"e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159"}
Nov 01 20:06:06.460530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.460482    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:06:06.463994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.463971    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.532139 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.532108    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 20:06:06.538565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.538528    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 20:06:06.547151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547042    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzb8m\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547090    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547194    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547238    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547270    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547296    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547326    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547352    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.547538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.547388    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.575419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.575392    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.648900    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzb8m\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.648948    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.648996    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649034    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649065    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649094    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649121    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649146    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649180    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.649624    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.650541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.650185    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.651186 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.651027    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.651186 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.651159    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.651418 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.651397    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.653033 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.652974    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.656008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.655984    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.713689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.713587    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.714509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.714423    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzb8m\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nzqvb\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.779185 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.777415    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"]
Nov 01 20:06:06.781247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.780985    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:06.781848 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.781832    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.841538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.838927    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"]
Nov 01 20:06:06.845600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.845437    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hwgb7\""
Nov 01 20:06:06.851540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.851300    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.851540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.851345    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.851979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.851834    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.851979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.851885    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.852576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.852224    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.852576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.852297    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.852576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.852322    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.852576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.852346    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.852576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.852387    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rf9q\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-kube-api-access-9rf9q\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953468    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953659 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953568    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953659 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953627    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953749 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953653    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953749 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953682    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953749 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953718    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rf9q\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-kube-api-access-9rf9q\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953754    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953784    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.953850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.953840    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.954280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.954257    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.954567 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.954546    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.956831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.955601    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.956831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.956233    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.956831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.956460    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.958519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.958422    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:06.959206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:06.959146    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:07.013537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.013281    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rf9q\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-kube-api-access-9rf9q\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:07.015205 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.015171    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc1fcc8-7c32-487d-b2d0-5b0645c43a26-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv\" (UID: \"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:07.114084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.113586    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:07.235935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.235830    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:06:07.520975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.520940    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"]
Nov 01 20:06:07.528930 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:07.528882    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc1fcc8_7c32_487d_b2d0_5b0645c43a26.slice/crio-cba71263a27b8c11ecf42bb3fc9bfab3ff452930f1cc5510ef1499a80b7a882f WatchSource:0}: Error finding container cba71263a27b8c11ecf42bb3fc9bfab3ff452930f1cc5510ef1499a80b7a882f: Status 404 returned error can't find the container with id cba71263a27b8c11ecf42bb3fc9bfab3ff452930f1cc5510ef1499a80b7a882f
Nov 01 20:06:07.770058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.769922    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerStarted","Data":"6179b4a4fc96d6ef1fab12328d18f7a736104862f374fbe320605b397bdff083"}
Nov 01 20:06:07.771991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:07.771841    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv" event={"ID":"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26","Type":"ContainerStarted","Data":"cba71263a27b8c11ecf42bb3fc9bfab3ff452930f1cc5510ef1499a80b7a882f"}
Nov 01 20:06:09.792473 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:09.792429    5066 generic.go:356] "Generic (PLEG): container finished" podID="867fc386-24da-497e-91ab-dd6ca920895b" containerID="d6bf6682995623b576b361cfd32b08142d31175632cc2c738c9d69c660b042f8" exitCode=0
Nov 01 20:06:09.792940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:09.792550    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerDied","Data":"d6bf6682995623b576b361cfd32b08142d31175632cc2c738c9d69c660b042f8"}
Nov 01 20:06:09.796086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:09.796059    5066 generic.go:356] "Generic (PLEG): container finished" podID="d43822ed-2c51-4bf7-999d-937564029b5a" containerID="321d32417e4c218fe80e8f7c1908111126932067c8797e7eb6246d0a4dcffcd0" exitCode=0
Nov 01 20:06:09.796243 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:09.796126    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg" event={"ID":"d43822ed-2c51-4bf7-999d-937564029b5a","Type":"ContainerDied","Data":"321d32417e4c218fe80e8f7c1908111126932067c8797e7eb6246d0a4dcffcd0"}
Nov 01 20:06:11.810053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:11.810013    5066 generic.go:356] "Generic (PLEG): container finished" podID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerID="8576a30e517a6bc1547c09cc2e1e886f90d286e310d74b278fa8fd1a9df21c3a" exitCode=0
Nov 01 20:06:11.810402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:11.810136    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerDied","Data":"8576a30e517a6bc1547c09cc2e1e886f90d286e310d74b278fa8fd1a9df21c3a"}
Nov 01 20:06:11.812387 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:11.812360    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerStarted","Data":"cd39c1c2b308690657c857ec10b7872ceed43c4f8f9c5eeb287cfde9834c0c41"}
Nov 01 20:06:11.815790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:11.815294    5066 generic.go:356] "Generic (PLEG): container finished" podID="045a4822-7409-4b83-8341-70f3005bc199" containerID="065baa85d05af2975cb16f7203d49a6a2bd8a464a7b630859dfc28323d91e14a" exitCode=0
Nov 01 20:06:11.815790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:11.815362    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerDied","Data":"065baa85d05af2975cb16f7203d49a6a2bd8a464a7b630859dfc28323d91e14a"}
Nov 01 20:06:12.654743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.654700    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.654855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.654810    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.654911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.654854    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.767814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.767776    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.767969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.767867    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.767969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.767920    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:06:12.821339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.821310    5066 generic.go:356] "Generic (PLEG): container finished" podID="867fc386-24da-497e-91ab-dd6ca920895b" containerID="f56009842669a3cb502dca357446fe0ee8cb6d22528811c1a30d6d4515960132" exitCode=0
Nov 01 20:06:12.821709 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.821384    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerDied","Data":"f56009842669a3cb502dca357446fe0ee8cb6d22528811c1a30d6d4515960132"}
Nov 01 20:06:12.823417 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.823395    5066 generic.go:356] "Generic (PLEG): container finished" podID="d43822ed-2c51-4bf7-999d-937564029b5a" containerID="abcab3669ac341c00d42ed61ca0963bb4642bd9c011e8a598df5b2601bf37b70" exitCode=0
Nov 01 20:06:12.823520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.823467    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg" event={"ID":"d43822ed-2c51-4bf7-999d-937564029b5a","Type":"ContainerDied","Data":"abcab3669ac341c00d42ed61ca0963bb4642bd9c011e8a598df5b2601bf37b70"}
Nov 01 20:06:12.825128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.825106    5066 generic.go:356] "Generic (PLEG): container finished" podID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerID="cd39c1c2b308690657c857ec10b7872ceed43c4f8f9c5eeb287cfde9834c0c41" exitCode=0
Nov 01 20:06:12.825233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.825174    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerDied","Data":"cd39c1c2b308690657c857ec10b7872ceed43c4f8f9c5eeb287cfde9834c0c41"}
Nov 01 20:06:12.826879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.826829    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerStarted","Data":"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"}
Nov 01 20:06:12.829822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.829799    5066 generic.go:356] "Generic (PLEG): container finished" podID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerID="3ea1fbbb36248cb3713b05bd245ef08c0e57700b1117c39a1a9e101581e9e39d" exitCode=0
Nov 01 20:06:12.829922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:12.829836    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerDied","Data":"3ea1fbbb36248cb3713b05bd245ef08c0e57700b1117c39a1a9e101581e9e39d"}
Nov 01 20:06:13.114490 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.114459    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:13.118850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.118826    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:13.165052 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.165004    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podStartSLOduration=1.7585849630000001 podStartE2EDuration="7.164990878s" totalImagesPullingTime="5.406405915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:13.163053162 +0000 UTC m=+226.547950752" watchObservedRunningTime="2026-11-01 20:06:13.164990878 +0000 UTC m=+226.549888471"
Nov 01 20:06:13.394580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.394524    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv" podStartSLOduration=2.160730733 podStartE2EDuration="7.394506245s" totalImagesPullingTime="5.233775512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:13.393782203 +0000 UTC m=+226.778679793" watchObservedRunningTime="2026-11-01 20:06:13.394506245 +0000 UTC m=+226.779403834"
Nov 01 20:06:13.782030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.781991    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:13.787930 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.787900    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:13.837827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.837787    5066 generic.go:356] "Generic (PLEG): container finished" podID="045a4822-7409-4b83-8341-70f3005bc199" containerID="e2fce75d5200f8133359dfd84deeafd23d3e844c7a4768d17c423a87e66eab9c" exitCode=0
Nov 01 20:06:13.838254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.837890    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerDied","Data":"e2fce75d5200f8133359dfd84deeafd23d3e844c7a4768d17c423a87e66eab9c"}
Nov 01 20:06:13.843776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.843715    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv" event={"ID":"5bc1fcc8-7c32-487d-b2d0-5b0645c43a26","Type":"ContainerStarted","Data":"0c67bbf25e7695b47d1134ba46baa67cde05ff90d5c9e62dc44f69c227c6b809"}
Nov 01 20:06:13.844420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.844371    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:13.844602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.844563    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:13.845658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.845638    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv"
Nov 01 20:06:13.845820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:13.845803    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:06:14.855441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:14.853708    5066 generic.go:356] "Generic (PLEG): container finished" podID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerID="ef5f084bbf75d59882c9c4cf462d63189a41902a3359e37fddf7f013ee20904a" exitCode=0
Nov 01 20:06:14.855441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:14.854893    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerDied","Data":"ef5f084bbf75d59882c9c4cf462d63189a41902a3359e37fddf7f013ee20904a"}
Nov 01 20:06:15.870517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:15.869790    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerStarted","Data":"5db59029d3b3d714f9dd04b63f015912cea2a577842de05f531f32b200be652e"}
Nov 01 20:06:16.878419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:16.878381    5066 generic.go:356] "Generic (PLEG): container finished" podID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerID="5db59029d3b3d714f9dd04b63f015912cea2a577842de05f531f32b200be652e" exitCode=0
Nov 01 20:06:16.878878 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:16.878433    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerDied","Data":"5db59029d3b3d714f9dd04b63f015912cea2a577842de05f531f32b200be652e"}
Nov 01 20:06:17.896560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:17.896242    5066 generic.go:356] "Generic (PLEG): container finished" podID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerID="cd8edec23d149798197b95141f74bd893955abda79fb240a09aea9fb4a53a5d4" exitCode=0
Nov 01 20:06:17.896560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:17.896361    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerDied","Data":"cd8edec23d149798197b95141f74bd893955abda79fb240a09aea9fb4a53a5d4"}
Nov 01 20:06:21.635992 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:06:21.635865    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:06:24.959346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:24.959309    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerStarted","Data":"e62000ba056cfcaf53619b0f67285d474e2371f9f3c9e94f8b470902ded038be"}
Nov 01 20:06:25.101149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.101047    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" podStartSLOduration=2.798961866 podStartE2EDuration="28.101031399s" totalImagesPullingTime="25.302069533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:25.100691669 +0000 UTC m=+238.485589261" watchObservedRunningTime="2026-11-01 20:06:25.101031399 +0000 UTC m=+238.485928987"
Nov 01 20:06:25.970171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.970095    5066 generic.go:356] "Generic (PLEG): container finished" podID="867fc386-24da-497e-91ab-dd6ca920895b" containerID="e62000ba056cfcaf53619b0f67285d474e2371f9f3c9e94f8b470902ded038be" exitCode=0
Nov 01 20:06:25.970583 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.970558    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerDied","Data":"e62000ba056cfcaf53619b0f67285d474e2371f9f3c9e94f8b470902ded038be"}
Nov 01 20:06:25.973609 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.973580    5066 generic.go:356] "Generic (PLEG): container finished" podID="d43822ed-2c51-4bf7-999d-937564029b5a" containerID="8c1f1d3307b24a825f5727cf5e1bd2d7008a80c92f6789704acdc871b4b9902f" exitCode=0
Nov 01 20:06:25.973723 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.973709    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg" event={"ID":"d43822ed-2c51-4bf7-999d-937564029b5a","Type":"ContainerDied","Data":"8c1f1d3307b24a825f5727cf5e1bd2d7008a80c92f6789704acdc871b4b9902f"}
Nov 01 20:06:25.976681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.976647    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerStarted","Data":"94363c7bb450806fa8cb5555ee628a0063bb73d69cf29e2fa5bcf05a3f16a60d"}
Nov 01 20:06:25.979124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.979098    5066 generic.go:356] "Generic (PLEG): container finished" podID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerID="fa5c31ff99152e38fcdcf6c1b8cb4e3eb75889ba40ce14647ffcf7b11ed97381" exitCode=0
Nov 01 20:06:25.979223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:25.979173    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerDied","Data":"fa5c31ff99152e38fcdcf6c1b8cb4e3eb75889ba40ce14647ffcf7b11ed97381"}
Nov 01 20:06:26.431189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:26.431125    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" podStartSLOduration=2.884748986 podStartE2EDuration="27.431106662s" totalImagesPullingTime="24.546357676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:26.430445422 +0000 UTC m=+239.815343011" watchObservedRunningTime="2026-11-01 20:06:26.431106662 +0000 UTC m=+239.816004254"
Nov 01 20:06:26.985686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:26.985656    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerStarted","Data":"312e95736e4255e99826be7322c3a9ff1d4ef6197ce7afbbf44d1d0b3ebb55ef"}
Nov 01 20:06:26.987734 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:26.987711    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerStarted","Data":"d010c2337c4563a4fb3c22410b40324d56fb76f96b3080dc89a89e2e0d076df8"}
Nov 01 20:06:26.989703 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:26.989676    5066 generic.go:356] "Generic (PLEG): container finished" podID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerID="94363c7bb450806fa8cb5555ee628a0063bb73d69cf29e2fa5bcf05a3f16a60d" exitCode=0
Nov 01 20:06:26.989794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:26.989718    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerDied","Data":"94363c7bb450806fa8cb5555ee628a0063bb73d69cf29e2fa5bcf05a3f16a60d"}
Nov 01 20:06:27.100558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.100448    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" podStartSLOduration=3.492615286 podStartE2EDuration="29.100430293s" totalImagesPullingTime="25.607815007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:27.099706488 +0000 UTC m=+240.484604079" watchObservedRunningTime="2026-11-01 20:06:27.100430293 +0000 UTC m=+240.485327884"
Nov 01 20:06:27.161413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.161388    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:06:27.165884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.165849    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\") pod \"d43822ed-2c51-4bf7-999d-937564029b5a\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") "
Nov 01 20:06:27.165993 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.165930    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\") pod \"d43822ed-2c51-4bf7-999d-937564029b5a\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") "
Nov 01 20:06:27.165993 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.165973    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\") pod \"d43822ed-2c51-4bf7-999d-937564029b5a\" (UID: \"d43822ed-2c51-4bf7-999d-937564029b5a\") "
Nov 01 20:06:27.166343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.166319    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle" pod "d43822ed-2c51-4bf7-999d-937564029b5a" (UID: "d43822ed-2c51-4bf7-999d-937564029b5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.168385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.168358    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k" pod "d43822ed-2c51-4bf7-999d-937564029b5a" (UID: "d43822ed-2c51-4bf7-999d-937564029b5a"). InnerVolumeSpecName "kube-api-access-svx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:27.171700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.171675    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util" pod "d43822ed-2c51-4bf7-999d-937564029b5a" (UID: "d43822ed-2c51-4bf7-999d-937564029b5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.183560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.183542    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:06:27.197666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.197647    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:27.214832 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.214790    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" podStartSLOduration=3.3947197940000002 podStartE2EDuration="30.214778308s" totalImagesPullingTime="26.820058514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:27.213295866 +0000 UTC m=+240.598193455" watchObservedRunningTime="2026-11-01 20:06:27.214778308 +0000 UTC m=+240.599675897"
Nov 01 20:06:27.266995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.266914    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\" (UniqueName: \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\") pod \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") "
Nov 01 20:06:27.266995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.266960    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\") pod \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") "
Nov 01 20:06:27.267218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267031    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\" (UniqueName: \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\") pod \"867fc386-24da-497e-91ab-dd6ca920895b\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") "
Nov 01 20:06:27.267218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267052    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\") pod \"867fc386-24da-497e-91ab-dd6ca920895b\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") "
Nov 01 20:06:27.267218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267089    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\") pod \"867fc386-24da-497e-91ab-dd6ca920895b\" (UID: \"867fc386-24da-497e-91ab-dd6ca920895b\") "
Nov 01 20:06:27.267218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267120    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\") pod \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\" (UID: \"cb16331f-29b9-43e0-a9b8-0d398d85ffb1\") "
Nov 01 20:06:27.267415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267359    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.267415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267380    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d43822ed-2c51-4bf7-999d-937564029b5a-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.267415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267394    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/d43822ed-2c51-4bf7-999d-937564029b5a-kube-api-access-svx2k\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.267594 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267548    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle" pod "cb16331f-29b9-43e0-a9b8-0d398d85ffb1" (UID: "cb16331f-29b9-43e0-a9b8-0d398d85ffb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.267651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.267596    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle" pod "867fc386-24da-497e-91ab-dd6ca920895b" (UID: "867fc386-24da-497e-91ab-dd6ca920895b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.269705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.269677    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6" pod "867fc386-24da-497e-91ab-dd6ca920895b" (UID: "867fc386-24da-497e-91ab-dd6ca920895b"). InnerVolumeSpecName "kube-api-access-rzrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:27.269705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.269694    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq" pod "cb16331f-29b9-43e0-a9b8-0d398d85ffb1" (UID: "cb16331f-29b9-43e0-a9b8-0d398d85ffb1"). InnerVolumeSpecName "kube-api-access-ql6kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:27.271268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.271247    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util" pod "867fc386-24da-497e-91ab-dd6ca920895b" (UID: "867fc386-24da-497e-91ab-dd6ca920895b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.271420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.271398    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util" pod "cb16331f-29b9-43e0-a9b8-0d398d85ffb1" (UID: "cb16331f-29b9-43e0-a9b8-0d398d85ffb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:27.367804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367771    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrz6\" (UniqueName: \"kubernetes.io/projected/867fc386-24da-497e-91ab-dd6ca920895b-kube-api-access-rzrz6\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.367804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367800    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.367804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367809    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/867fc386-24da-497e-91ab-dd6ca920895b-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.368034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367818    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.368034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367827    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6kq\" (UniqueName: \"kubernetes.io/projected/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-kube-api-access-ql6kq\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.368034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.367835    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb16331f-29b9-43e0-a9b8-0d398d85ffb1-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:27.996299 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.996140    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg"
Nov 01 20:06:27.996299 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.996170    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mdxzg" event={"ID":"cb16331f-29b9-43e0-a9b8-0d398d85ffb1","Type":"ContainerDied","Data":"e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159"}
Nov 01 20:06:27.996299 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.996212    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42d15829364ec151335b982d2e0ab7840f27186cf59fcb8069a4a72fbd21159"
Nov 01 20:06:27.998416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.998392    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q"
Nov 01 20:06:27.998535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.998403    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rg59q" event={"ID":"867fc386-24da-497e-91ab-dd6ca920895b","Type":"ContainerDied","Data":"fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b"}
Nov 01 20:06:27.998535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:27.998451    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4e9c22a5f47162e64f64abf87939b6a6f33eecd15cc7cd38654c241ae9f94b"
Nov 01 20:06:28.000674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.000648    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg" event={"ID":"d43822ed-2c51-4bf7-999d-937564029b5a","Type":"ContainerDied","Data":"5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302"}
Nov 01 20:06:28.000674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.000659    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdmvgg"
Nov 01 20:06:28.000812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.000683    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bda43128a3ae44ef06c7d98f080b9fb7de1c158c3162fc186d6ad4018ff9302"
Nov 01 20:06:28.006182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.005853    5066 generic.go:356] "Generic (PLEG): container finished" podID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerID="312e95736e4255e99826be7322c3a9ff1d4ef6197ce7afbbf44d1d0b3ebb55ef" exitCode=0
Nov 01 20:06:28.006182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.005937    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerDied","Data":"312e95736e4255e99826be7322c3a9ff1d4ef6197ce7afbbf44d1d0b3ebb55ef"}
Nov 01 20:06:28.008179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.008151    5066 generic.go:356] "Generic (PLEG): container finished" podID="045a4822-7409-4b83-8341-70f3005bc199" containerID="d010c2337c4563a4fb3c22410b40324d56fb76f96b3080dc89a89e2e0d076df8" exitCode=0
Nov 01 20:06:28.008269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.008227    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerDied","Data":"d010c2337c4563a4fb3c22410b40324d56fb76f96b3080dc89a89e2e0d076df8"}
Nov 01 20:06:28.145390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.145366    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:06:28.173552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.173528    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\") pod \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") "
Nov 01 20:06:28.173673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.173559    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\" (UniqueName: \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\") pod \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") "
Nov 01 20:06:28.173673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.173596    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\") pod \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\" (UID: \"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8\") "
Nov 01 20:06:28.174320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.174294    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle" pod "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" (UID: "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:28.175564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.175543    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n" pod "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" (UID: "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8"). InnerVolumeSpecName "kube-api-access-2dr4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:28.178983 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.178959    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util" pod "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" (UID: "06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:28.274819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.274763    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:28.274819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.274788    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dr4n\" (UniqueName: \"kubernetes.io/projected/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-kube-api-access-2dr4n\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:28.274819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:28.274798    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.014891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.014813    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx"
Nov 01 20:06:29.014891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.014836    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44mmx" event={"ID":"06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8","Type":"ContainerDied","Data":"d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984"}
Nov 01 20:06:29.014891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.014881    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d834b7887fa4c1e5450493015fc2de54d2bb33cecbf02ddff2dfa4dda2d72984"
Nov 01 20:06:29.162663 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.162640    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:06:29.181340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.181317    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\") pod \"045a4822-7409-4b83-8341-70f3005bc199\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") "
Nov 01 20:06:29.181472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.181394    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\") pod \"045a4822-7409-4b83-8341-70f3005bc199\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") "
Nov 01 20:06:29.181553 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.181476    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\" (UniqueName: \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\") pod \"045a4822-7409-4b83-8341-70f3005bc199\" (UID: \"045a4822-7409-4b83-8341-70f3005bc199\") "
Nov 01 20:06:29.182003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.181979    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle" pod "045a4822-7409-4b83-8341-70f3005bc199" (UID: "045a4822-7409-4b83-8341-70f3005bc199"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:29.183304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.183276    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn" pod "045a4822-7409-4b83-8341-70f3005bc199" (UID: "045a4822-7409-4b83-8341-70f3005bc199"). InnerVolumeSpecName "kube-api-access-ldlmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:29.187307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.187282    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util" pod "045a4822-7409-4b83-8341-70f3005bc199" (UID: "045a4822-7409-4b83-8341-70f3005bc199"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:29.187961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.187938    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:06:29.282652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282576    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\" (UniqueName: \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\") pod \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") "
Nov 01 20:06:29.282652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282626    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\") pod \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") "
Nov 01 20:06:29.282652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282651    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\") pod \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\" (UID: \"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1\") "
Nov 01 20:06:29.282894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282815    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldlmn\" (UniqueName: \"kubernetes.io/projected/045a4822-7409-4b83-8341-70f3005bc199-kube-api-access-ldlmn\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.282894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282833    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.282894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.282847    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/045a4822-7409-4b83-8341-70f3005bc199-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.283374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.283341    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle" pod "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" (UID: "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:29.285012 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.284987    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7" pod "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" (UID: "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1"). InnerVolumeSpecName "kube-api-access-sf7l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:29.288606 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.288581    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util" pod "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" (UID: "7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:06:29.383758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.383722    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf7l7\" (UniqueName: \"kubernetes.io/projected/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-kube-api-access-sf7l7\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.383758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.383756    5066 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-util\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:29.383758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:29.383766    5066 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:06:30.020761 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.020732    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb"
Nov 01 20:06:30.021231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.020728    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714x4nb" event={"ID":"7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1","Type":"ContainerDied","Data":"ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779"}
Nov 01 20:06:30.021231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.020850    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba42936402314c08d4342777b783ce678843777081a913f22402b6953bd01779"
Nov 01 20:06:30.022658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.022636    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg" event={"ID":"045a4822-7409-4b83-8341-70f3005bc199","Type":"ContainerDied","Data":"3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e"}
Nov 01 20:06:30.022798 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.022667    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f07cf8e0df45f6dd039105bad5cd88d6b9c7082dec3edfb9e12c5efd212c50e"
Nov 01 20:06:30.022798 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:30.022671    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashtvg"
Nov 01 20:06:38.036554 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036517    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"]
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036852    5066 cpu_manager.go:414] "containerMap: removing container" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036863    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036874    5066 cpu_manager.go:414] "containerMap: removing container" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="pull"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036879    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="pull"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036884    5066 cpu_manager.go:414] "containerMap: removing container" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036889    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036896    5066 cpu_manager.go:414] "containerMap: removing container" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036900    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036908    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036913    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036919    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036923    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036930    5066 cpu_manager.go:414] "containerMap: removing container" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036934    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="extract"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036941    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="util"
Nov 01 20:06:38.036938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036945    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="util"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036952    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036956    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036964    5066 cpu_manager.go:414] "containerMap: removing container" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036968    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036975    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036979    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036985    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036988    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.036997    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037000    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037011    5066 cpu_manager.go:414] "containerMap: removing container" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="util"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037015    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="util"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037043    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="util"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037048    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="util"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037055    5066 cpu_manager.go:414] "containerMap: removing container" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037059    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="pull"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037066    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037071    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037077    5066 cpu_manager.go:414] "containerMap: removing container" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037082    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037136    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca761b7-6e22-4c33-a5a1-0c30e2dcd9d1" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037144    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="867fc386-24da-497e-91ab-dd6ca920895b" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037148    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="06685dbf-0ee3-4ace-b8cf-8cf4afa98cc8" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037155    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43822ed-2c51-4bf7-999d-937564029b5a" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037160    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="045a4822-7409-4b83-8341-70f3005bc199" containerName="extract"
Nov 01 20:06:38.037379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.037166    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb16331f-29b9-43e0-a9b8-0d398d85ffb1" containerName="extract"
Nov 01 20:06:38.039029 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.039013    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.146416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.146386    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"]
Nov 01 20:06:38.157325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157298    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.157448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157349    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.157448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157433    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkqj\" (UniqueName: \"kubernetes.io/projected/3d52bf76-fa3b-445b-99f4-720495f9b785-kube-api-access-7dkqj\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.157581 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157480    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.157581 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157525    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.157658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.157642    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.194343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194308    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 20:06:38.194343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194335    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 20:06:38.194343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194338    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 20:06:38.194615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194312    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 20:06:38.194615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194372    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 20:06:38.194615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.194395    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\""
Nov 01 20:06:38.258535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258506    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.258720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258558    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.258720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258586    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkqj\" (UniqueName: \"kubernetes.io/projected/3d52bf76-fa3b-445b-99f4-720495f9b785-kube-api-access-7dkqj\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.258720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258628    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.258720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258660    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.258720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.258703    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.259106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.259079    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.260757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.260728    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d52bf76-fa3b-445b-99f4-720495f9b785-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.261442 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.261421    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.261442 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.261431    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.261599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.261479    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d52bf76-fa3b-445b-99f4-720495f9b785-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.315582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.315514    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkqj\" (UniqueName: \"kubernetes.io/projected/3d52bf76-fa3b-445b-99f4-720495f9b785-kube-api-access-7dkqj\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-kwskb\" (UID: \"3d52bf76-fa3b-445b-99f4-720495f9b785\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.350506 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.350472    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:38.857067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:38.857035    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"]
Nov 01 20:06:38.857265 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:38.857246    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d52bf76_fa3b_445b_99f4_720495f9b785.slice/crio-e19bec3ab9aa443c8e440997636ae684b3ed9b0c2abbb7e6954609d992c9a403 WatchSource:0}: Error finding container e19bec3ab9aa443c8e440997636ae684b3ed9b0c2abbb7e6954609d992c9a403: Status 404 returned error can't find the container with id e19bec3ab9aa443c8e440997636ae684b3ed9b0c2abbb7e6954609d992c9a403
Nov 01 20:06:39.065815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:39.065783    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb" event={"ID":"3d52bf76-fa3b-445b-99f4-720495f9b785","Type":"ContainerStarted","Data":"e19bec3ab9aa443c8e440997636ae684b3ed9b0c2abbb7e6954609d992c9a403"}
Nov 01 20:06:42.344536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.344481    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"]
Nov 01 20:06:42.346909 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.346887    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:42.394897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.394870    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh92\" (UniqueName: \"kubernetes.io/projected/631dad46-02be-4fba-a072-43be25b7a0e5-kube-api-access-4dh92\") pod \"authorino-operator-7b7bb4bc77-4pll5\" (UID: \"631dad46-02be-4fba-a072-43be25b7a0e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:42.398840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.398813    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 20:06:42.398964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.398895    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 20:06:42.448571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.448538    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jjkzl\""
Nov 01 20:06:42.450264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.450231    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"]
Nov 01 20:06:42.495532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.495481    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh92\" (UniqueName: \"kubernetes.io/projected/631dad46-02be-4fba-a072-43be25b7a0e5-kube-api-access-4dh92\") pod \"authorino-operator-7b7bb4bc77-4pll5\" (UID: \"631dad46-02be-4fba-a072-43be25b7a0e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:42.555253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.555221    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh92\" (UniqueName: \"kubernetes.io/projected/631dad46-02be-4fba-a072-43be25b7a0e5-kube-api-access-4dh92\") pod \"authorino-operator-7b7bb4bc77-4pll5\" (UID: \"631dad46-02be-4fba-a072-43be25b7a0e5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:42.660289 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:42.660211    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:43.045971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:43.045947    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"]
Nov 01 20:06:43.048078 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:43.048051    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631dad46_02be_4fba_a072_43be25b7a0e5.slice/crio-1e20d8fe880ff03b903387612fcac9c529d07d72440d87f3c5220c76a4068881 WatchSource:0}: Error finding container 1e20d8fe880ff03b903387612fcac9c529d07d72440d87f3c5220c76a4068881: Status 404 returned error can't find the container with id 1e20d8fe880ff03b903387612fcac9c529d07d72440d87f3c5220c76a4068881
Nov 01 20:06:43.092392 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:43.092362    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5" event={"ID":"631dad46-02be-4fba-a072-43be25b7a0e5","Type":"ContainerStarted","Data":"1e20d8fe880ff03b903387612fcac9c529d07d72440d87f3c5220c76a4068881"}
Nov 01 20:06:43.093829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:43.093807    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb" event={"ID":"3d52bf76-fa3b-445b-99f4-720495f9b785","Type":"ContainerStarted","Data":"ccd34bbf39d887eba0248b0760c2901a9f0ad3d4ced69ffc592e326d95ab271d"}
Nov 01 20:06:43.952050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:43.951975    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"]
Nov 01 20:06:43.954049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:43.954032    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.005858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.005831    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:06:44.006007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.005993    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 20:06:44.006279 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.006264    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bthm5\""
Nov 01 20:06:44.007583 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.007566    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744df5c8-76a0-4685-b90e-76301db358d7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.007646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.007631    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznvz\" (UniqueName: \"kubernetes.io/projected/744df5c8-76a0-4685-b90e-76301db358d7-kube-api-access-fznvz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.016512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.016478    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"]
Nov 01 20:06:44.099799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.099762    5066 generic.go:356] "Generic (PLEG): container finished" podID="3d52bf76-fa3b-445b-99f4-720495f9b785" containerID="ccd34bbf39d887eba0248b0760c2901a9f0ad3d4ced69ffc592e326d95ab271d" exitCode=0
Nov 01 20:06:44.099974 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.099835    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb" event={"ID":"3d52bf76-fa3b-445b-99f4-720495f9b785","Type":"ContainerDied","Data":"ccd34bbf39d887eba0248b0760c2901a9f0ad3d4ced69ffc592e326d95ab271d"}
Nov 01 20:06:44.108409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.108384    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fznvz\" (UniqueName: \"kubernetes.io/projected/744df5c8-76a0-4685-b90e-76301db358d7-kube-api-access-fznvz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.108470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.108432    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744df5c8-76a0-4685-b90e-76301db358d7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.108804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.108787    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744df5c8-76a0-4685-b90e-76301db358d7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.165953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.165927    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznvz\" (UniqueName: \"kubernetes.io/projected/744df5c8-76a0-4685-b90e-76301db358d7-kube-api-access-fznvz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dj27w\" (UID: \"744df5c8-76a0-4685-b90e-76301db358d7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.264196 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.264166    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"
Nov 01 20:06:44.687785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:44.687699    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w"]
Nov 01 20:06:44.690584 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:44.690554    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744df5c8_76a0_4685_b90e_76301db358d7.slice/crio-0c0c1d9b4eda442f7b7182a36e44f7692b269138db0562e14726043b5568a596 WatchSource:0}: Error finding container 0c0c1d9b4eda442f7b7182a36e44f7692b269138db0562e14726043b5568a596: Status 404 returned error can't find the container with id 0c0c1d9b4eda442f7b7182a36e44f7692b269138db0562e14726043b5568a596
Nov 01 20:06:45.106534 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:45.106483    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb" event={"ID":"3d52bf76-fa3b-445b-99f4-720495f9b785","Type":"ContainerStarted","Data":"f637b0ee0012d1609168aa5cb9b325e8e0ea7f9696e42cafac3324c43411f7d9"}
Nov 01 20:06:45.106972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:45.106699    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:45.107968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:45.107939    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w" event={"ID":"744df5c8-76a0-4685-b90e-76301db358d7","Type":"ContainerStarted","Data":"0c0c1d9b4eda442f7b7182a36e44f7692b269138db0562e14726043b5568a596"}
Nov 01 20:06:45.253366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:45.253299    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb" podStartSLOduration=3.10142403 podStartE2EDuration="8.253278161s" totalImagesPullingTime="4.684572719s" totalInitContainerRuntime="467.281412ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:45.248149131 +0000 UTC m=+258.633046721" watchObservedRunningTime="2026-11-01 20:06:45.253278161 +0000 UTC m=+258.638175755"
Nov 01 20:06:48.127272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:48.127176    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w" event={"ID":"744df5c8-76a0-4685-b90e-76301db358d7","Type":"ContainerStarted","Data":"3e32c84fc835c3dab19465c0474b7f3a5f1bba68002186ba207cf63e889c7e4b"}
Nov 01 20:06:48.128869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:48.128844    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5" event={"ID":"631dad46-02be-4fba-a072-43be25b7a0e5","Type":"ContainerStarted","Data":"49319e63c447e162723ddabe717e1240bcd726c42f4a13d6dd0f850d034781c1"}
Nov 01 20:06:48.128996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:48.128957    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:06:48.347746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:48.347697    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dj27w" podStartSLOduration=2.588807692 podStartE2EDuration="5.347684021s" totalImagesPullingTime="2.758876329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:48.345700268 +0000 UTC m=+261.730597857" watchObservedRunningTime="2026-11-01 20:06:48.347684021 +0000 UTC m=+261.732581610"
Nov 01 20:06:48.466915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:48.466863    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5" podStartSLOduration=1.718904092 podStartE2EDuration="6.466851717s" totalImagesPullingTime="4.747947625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:48.464648743 +0000 UTC m=+261.849546334" watchObservedRunningTime="2026-11-01 20:06:48.466851717 +0000 UTC m=+261.851749305"
Nov 01 20:06:50.728635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.728604    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 20:06:50.735798 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.735778    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:50.779400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.779370    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 20:06:50.826759 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.826728    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4n5t\""
Nov 01 20:06:50.870181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.870149    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pvh\" (UniqueName: \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:50.870349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.870231    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:50.971784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.971745    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:50.971967 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.971818    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pvh\" (UniqueName: \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:50.972161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:50.972138    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:51.039709 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:51.039633    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pvh\" (UniqueName: \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\") pod \"kuadrant-operator-controller-manager-678d87766f-mjh7s\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:51.048373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:51.048350    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:06:51.481295 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:51.481257    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 20:06:51.483205 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:06:51.483173    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da5efcb_cdd3_48d5_813d_17bc1d59717c.slice/crio-236fe54ceaa79b3f9bb446317d83cc1feab6c71e36d9233745b17d39c99a72cb WatchSource:0}: Error finding container 236fe54ceaa79b3f9bb446317d83cc1feab6c71e36d9233745b17d39c99a72cb: Status 404 returned error can't find the container with id 236fe54ceaa79b3f9bb446317d83cc1feab6c71e36d9233745b17d39c99a72cb
Nov 01 20:06:51.627776 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:06:51.627735    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:06:52.148154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:52.148117    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" event={"ID":"0da5efcb-cdd3-48d5-813d-17bc1d59717c","Type":"ContainerStarted","Data":"236fe54ceaa79b3f9bb446317d83cc1feab6c71e36d9233745b17d39c99a72cb"}
Nov 01 20:06:56.115939 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:56.115907    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-kwskb"
Nov 01 20:06:59.136154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:06:59.136101    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4pll5"
Nov 01 20:07:00.189441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:00.189399    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" event={"ID":"0da5efcb-cdd3-48d5-813d-17bc1d59717c","Type":"ContainerStarted","Data":"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"}
Nov 01 20:07:00.189950 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:00.189551    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:07:00.411706 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:00.411648    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" podStartSLOduration=2.713425713 podStartE2EDuration="10.411632116s" totalImagesPullingTime="7.698206403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:00.360834199 +0000 UTC m=+273.745731791" watchObservedRunningTime="2026-11-01 20:07:00.411632116 +0000 UTC m=+273.796529705"
Nov 01 20:07:09.167162 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.167121    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"]
Nov 01 20:07:09.169376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.169358    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.221051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.221021    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nw87d\""
Nov 01 20:07:09.272863 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.272826    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"]
Nov 01 20:07:09.342911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.342880    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-kube-api-access-dh4lb\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.342927    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb53d414-7ae6-4124-be91-0552575a3cfd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.342952    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343081    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343121    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343230    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343255    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343277    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.343349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.343296    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444309    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb53d414-7ae6-4124-be91-0552575a3cfd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444350    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444372    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444427    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444572    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444599    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444629    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444655    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444717    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-kube-api-access-dh4lb\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.444968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444902    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.445080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.444971    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.445080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.445061    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.445360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.445329    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb53d414-7ae6-4124-be91-0552575a3cfd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.445535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.445377    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.446834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.446818    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.447231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.447214    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.513757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.513726    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.516742 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.516718    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/fb53d414-7ae6-4124-be91-0552575a3cfd-kube-api-access-dh4lb\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv\" (UID: \"fb53d414-7ae6-4124-be91-0552575a3cfd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:09.780474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:09.780436    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:10.207647 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:10.207569    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"]
Nov 01 20:07:10.247552 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:07:10.247525    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb53d414_7ae6_4124_be91_0552575a3cfd.slice/crio-63919fd8494f1f197819028a9cf2e9e9f4eb958e77f2e14ea1c3410a138d2e22 WatchSource:0}: Error finding container 63919fd8494f1f197819028a9cf2e9e9f4eb958e77f2e14ea1c3410a138d2e22: Status 404 returned error can't find the container with id 63919fd8494f1f197819028a9cf2e9e9f4eb958e77f2e14ea1c3410a138d2e22
Nov 01 20:07:10.249524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:10.249471    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:07:10.249600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:10.249561    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:07:10.249600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:10.249590    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:07:11.096393 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.096360    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:11.098770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.098754    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.154276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.154241    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z5b7z\""
Nov 01 20:07:11.158104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.158068    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg87x\" (UniqueName: \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.158272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.158229    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.196410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.196385    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 20:07:11.203998 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.203974    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 20:07:11.206014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.205993    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:11.249306 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.249270    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv" event={"ID":"fb53d414-7ae6-4124-be91-0552575a3cfd","Type":"ContainerStarted","Data":"d2d60d1173f7d5757f62a875421a7fc2e4d958b6473e9694793f99b616395f7b"}
Nov 01 20:07:11.249666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.249328    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv" event={"ID":"fb53d414-7ae6-4124-be91-0552575a3cfd","Type":"ContainerStarted","Data":"63919fd8494f1f197819028a9cf2e9e9f4eb958e77f2e14ea1c3410a138d2e22"}
Nov 01 20:07:11.258890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.258870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.258960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.258916    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg87x\" (UniqueName: \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.259558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.259540    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.317900 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.317868    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg87x\" (UniqueName: \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\") pod \"limitador-limitador-74dfcfb894-lv47b\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.345131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.345102    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:11.387618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.387528    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv" podStartSLOduration=2.387485844 podStartE2EDuration="2.387485844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:11.385506701 +0000 UTC m=+284.770404285" watchObservedRunningTime="2026-11-01 20:07:11.387485844 +0000 UTC m=+284.772383434"
Nov 01 20:07:11.411118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.411096    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:11.780727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.780693    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:11.785509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.785470    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:11.795816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:11.791815    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:11.795816 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:07:11.793721    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4771dbb1_e484_4f6b_93fa_08f639511dd1.slice/crio-b240274970130ab281a045e1b76ccbe24c710ce5d02ab50ec5e07e08bed08963 WatchSource:0}: Error finding container b240274970130ab281a045e1b76ccbe24c710ce5d02ab50ec5e07e08bed08963: Status 404 returned error can't find the container with id b240274970130ab281a045e1b76ccbe24c710ce5d02ab50ec5e07e08bed08963
Nov 01 20:07:12.255428 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:12.255392    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" event={"ID":"4771dbb1-e484-4f6b-93fa-08f639511dd1","Type":"ContainerStarted","Data":"b240274970130ab281a045e1b76ccbe24c710ce5d02ab50ec5e07e08bed08963"}
Nov 01 20:07:12.255823 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:12.255678    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:12.256605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:12.256590    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv"
Nov 01 20:07:15.279555 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:15.279512    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" event={"ID":"4771dbb1-e484-4f6b-93fa-08f639511dd1","Type":"ContainerStarted","Data":"12bc0f176bebe07261cef7cdd97bb6a96a2b7c62480d87763c74b57b2b50ae8d"}
Nov 01 20:07:15.279941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:15.279579    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:15.393230 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:15.393134    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" podStartSLOduration=1.950264035 podStartE2EDuration="4.393119079s" totalImagesPullingTime="2.442855044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:15.392847244 +0000 UTC m=+288.777744836" watchObservedRunningTime="2026-11-01 20:07:15.393119079 +0000 UTC m=+288.778016683"
Nov 01 20:07:21.061563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.061529    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"]
Nov 01 20:07:21.066114 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.066086    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.120433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.120400    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 20:07:21.133971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.133944    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dlc\" (UniqueName: \"kubernetes.io/projected/be0efee3-c422-4d59-a0b7-67c656df942c-kube-api-access-89dlc\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.134122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.133981    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.134122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.134021    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0efee3-c422-4d59-a0b7-67c656df942c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.134199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.134174    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.169782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.169754    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 20:07:21.170020 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.170001    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-np7zt\""
Nov 01 20:07:21.171786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.171763    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"]
Nov 01 20:07:21.234776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.234746    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.234922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.234791    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dlc\" (UniqueName: \"kubernetes.io/projected/be0efee3-c422-4d59-a0b7-67c656df942c-kube-api-access-89dlc\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.234922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.234908    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.235002 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.234958    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0efee3-c422-4d59-a0b7-67c656df942c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.235236 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.235217    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.235276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.235218    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be0efee3-c422-4d59-a0b7-67c656df942c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.269838 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.269813    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 20:07:21.276470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.276450    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0efee3-c422-4d59-a0b7-67c656df942c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.293210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.293186    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dlc\" (UniqueName: \"kubernetes.io/projected/be0efee3-c422-4d59-a0b7-67c656df942c-kube-api-access-89dlc\") pod \"kserve-module-controller-manager-7566cf5fc8-cmbpq\" (UID: \"be0efee3-c422-4d59-a0b7-67c656df942c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.377828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.377764    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:21.627628 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:07:21.627589    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:07:21.795067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:21.795041    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"]
Nov 01 20:07:21.797207 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:07:21.797177    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0efee3_c422_4d59_a0b7_67c656df942c.slice/crio-c766b3067333b60ea8d26f287044ea1b025db63337467789897d4890165e5408 WatchSource:0}: Error finding container c766b3067333b60ea8d26f287044ea1b025db63337467789897d4890165e5408: Status 404 returned error can't find the container with id c766b3067333b60ea8d26f287044ea1b025db63337467789897d4890165e5408
Nov 01 20:07:22.314711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:22.314673    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq" event={"ID":"be0efee3-c422-4d59-a0b7-67c656df942c","Type":"ContainerStarted","Data":"c766b3067333b60ea8d26f287044ea1b025db63337467789897d4890165e5408"}
Nov 01 20:07:25.331776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:25.331742    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq" event={"ID":"be0efee3-c422-4d59-a0b7-67c656df942c","Type":"ContainerStarted","Data":"506465d8fe564608b5254f1d23e3c5cb7249560283f60dd87aa1071486376861"}
Nov 01 20:07:25.332204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:25.331916    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:25.498573 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:25.498513    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq" podStartSLOduration=1.588484937 podStartE2EDuration="4.498484456s" totalImagesPullingTime="2.909999519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:25.496134754 +0000 UTC m=+298.881032346" watchObservedRunningTime="2026-11-01 20:07:25.498484456 +0000 UTC m=+298.883382107"
Nov 01 20:07:26.286088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:26.286054    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:26.754073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:26.754040    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:07:26.755374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:26.755349    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:07:29.829218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:29.829176    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:29.829656 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:29.829401    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" podUID="4771dbb1-e484-4f6b-93fa-08f639511dd1" containerName="limitador" containerID="cri-o://12bc0f176bebe07261cef7cdd97bb6a96a2b7c62480d87763c74b57b2b50ae8d" gracePeriod=30
Nov 01 20:07:30.358458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.358424    5066 generic.go:356] "Generic (PLEG): container finished" podID="4771dbb1-e484-4f6b-93fa-08f639511dd1" containerID="12bc0f176bebe07261cef7cdd97bb6a96a2b7c62480d87763c74b57b2b50ae8d" exitCode=0
Nov 01 20:07:30.358607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.358518    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" event={"ID":"4771dbb1-e484-4f6b-93fa-08f639511dd1","Type":"ContainerDied","Data":"12bc0f176bebe07261cef7cdd97bb6a96a2b7c62480d87763c74b57b2b50ae8d"}
Nov 01 20:07:30.373233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.373208    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:30.417650 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.417575    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\" (UniqueName: \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\") pod \"4771dbb1-e484-4f6b-93fa-08f639511dd1\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") "
Nov 01 20:07:30.417797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.417732    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\" (UniqueName: \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\") pod \"4771dbb1-e484-4f6b-93fa-08f639511dd1\" (UID: \"4771dbb1-e484-4f6b-93fa-08f639511dd1\") "
Nov 01 20:07:30.418054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.418033    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file" pod "4771dbb1-e484-4f6b-93fa-08f639511dd1" (UID: "4771dbb1-e484-4f6b-93fa-08f639511dd1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:07:30.419784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.419763    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x" pod "4771dbb1-e484-4f6b-93fa-08f639511dd1" (UID: "4771dbb1-e484-4f6b-93fa-08f639511dd1"). InnerVolumeSpecName "kube-api-access-qg87x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:07:30.519220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.519179    5066 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4771dbb1-e484-4f6b-93fa-08f639511dd1-config-file\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:07:30.519220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:30.519220    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg87x\" (UniqueName: \"kubernetes.io/projected/4771dbb1-e484-4f6b-93fa-08f639511dd1-kube-api-access-qg87x\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:07:31.364763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.364731    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b"
Nov 01 20:07:31.365205 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.364775    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lv47b" event={"ID":"4771dbb1-e484-4f6b-93fa-08f639511dd1","Type":"ContainerDied","Data":"b240274970130ab281a045e1b76ccbe24c710ce5d02ab50ec5e07e08bed08963"}
Nov 01 20:07:31.365205 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.364842    5066 scope.go:122] "RemoveContainer" containerID="12bc0f176bebe07261cef7cdd97bb6a96a2b7c62480d87763c74b57b2b50ae8d"
Nov 01 20:07:31.406945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.406912    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"]
Nov 01 20:07:31.407308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.407295    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4771dbb1-e484-4f6b-93fa-08f639511dd1" containerName="limitador"
Nov 01 20:07:31.407363 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.407309    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4771dbb1-e484-4f6b-93fa-08f639511dd1" containerName="limitador"
Nov 01 20:07:31.407402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.407377    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4771dbb1-e484-4f6b-93fa-08f639511dd1" containerName="limitador"
Nov 01 20:07:31.434961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.434931    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.468442 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.468404    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"]
Nov 01 20:07:31.486975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.486948    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 20:07:31.487156 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.487015    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 20:07:31.487156 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.487115    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 20:07:31.504824 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.504798    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 20:07:31.514638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.514609    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4s8m4\""
Nov 01 20:07:31.514638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.514633    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 20:07:31.529158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.529124    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70fcc861-99d9-4b67-b2f1-1f45665d27f9-manager-config\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.529308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.529179    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-cert\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.529308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.529259    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-metrics-certs\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.529384 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.529346    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqjm\" (UniqueName: \"kubernetes.io/projected/70fcc861-99d9-4b67-b2f1-1f45665d27f9-kube-api-access-lcqjm\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.573125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.573087    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:31.576627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.576598    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lv47b"]
Nov 01 20:07:31.630461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.630389    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqjm\" (UniqueName: \"kubernetes.io/projected/70fcc861-99d9-4b67-b2f1-1f45665d27f9-kube-api-access-lcqjm\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.630461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.630423    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70fcc861-99d9-4b67-b2f1-1f45665d27f9-manager-config\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.630461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.630461    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-cert\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.630705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.630488    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-metrics-certs\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.631181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.631160    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70fcc861-99d9-4b67-b2f1-1f45665d27f9-manager-config\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.632929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.632910    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-cert\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.633077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.633060    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fcc861-99d9-4b67-b2f1-1f45665d27f9-metrics-certs\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.690479 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.690447    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqjm\" (UniqueName: \"kubernetes.io/projected/70fcc861-99d9-4b67-b2f1-1f45665d27f9-kube-api-access-lcqjm\") pod \"jobset-controller-manager-d7c7c6b5b-kw94g\" (UID: \"70fcc861-99d9-4b67-b2f1-1f45665d27f9\") " pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:31.757268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:31.757233    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:32.140317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:32.140292    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"]
Nov 01 20:07:32.142235 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:07:32.142206    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fcc861_99d9_4b67_b2f1_1f45665d27f9.slice/crio-f98ea6160b3ab4654775f54eeb78b8c4affe76139103ffa77a720270f9d3cab1 WatchSource:0}: Error finding container f98ea6160b3ab4654775f54eeb78b8c4affe76139103ffa77a720270f9d3cab1: Status 404 returned error can't find the container with id f98ea6160b3ab4654775f54eeb78b8c4affe76139103ffa77a720270f9d3cab1
Nov 01 20:07:32.144145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:32.144128    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:07:32.375170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:32.375133    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g" event={"ID":"70fcc861-99d9-4b67-b2f1-1f45665d27f9","Type":"ContainerStarted","Data":"f98ea6160b3ab4654775f54eeb78b8c4affe76139103ffa77a720270f9d3cab1"}
Nov 01 20:07:32.729334 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:32.729297    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4771dbb1-e484-4f6b-93fa-08f639511dd1" path="/var/lib/kubelet/pods/4771dbb1-e484-4f6b-93fa-08f639511dd1/volumes"
Nov 01 20:07:37.406216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:37.406179    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g" event={"ID":"70fcc861-99d9-4b67-b2f1-1f45665d27f9","Type":"ContainerStarted","Data":"e5f85fc72e43768231c335f84031257657bff4c232ae3390c85fc941763c7ac8"}
Nov 01 20:07:37.406721 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:37.406303    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:37.520558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:37.520483    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g" podStartSLOduration=1.783691905 podStartE2EDuration="6.520465153s" totalImagesPullingTime="4.736773248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:37.519521351 +0000 UTC m=+310.904418939" watchObservedRunningTime="2026-11-01 20:07:37.520465153 +0000 UTC m=+310.905362744"
Nov 01 20:07:46.338428 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:46.338346    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cmbpq"
Nov 01 20:07:48.417227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:07:48.417197    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-d7c7c6b5b-kw94g"
Nov 01 20:07:51.627855 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:07:51.627809    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:08:21.628409 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:08:21.628371    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:08:51.628059 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:08:51.628019    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:08:58.954639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:58.954588    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:37406->10.132.0.37:15021: read: connection reset by peer" start-of-body=
Nov 01 20:08:58.955005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:58.954664    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:37406->10.132.0.37:15021: read: connection reset by peer"
Nov 01 20:08:59.833561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.833528    5066 generic.go:356] "Generic (PLEG): container finished" podID="154811b0-d204-4fa5-96b4-8f648e70292d" containerID="e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898" exitCode=0
Nov 01 20:08:59.833726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.833583    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerDied","Data":"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"}
Nov 01 20:08:59.833902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.833887    5066 scope.go:122] "RemoveContainer" containerID="e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"
Nov 01 20:08:59.834918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.834887    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:08:59.834989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.834952    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:08:59.834989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:08:59.834985    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:09:00.840751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:00.840715    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerStarted","Data":"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"}
Nov 01 20:09:01.784936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:01.784850    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:09:01.785588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:01.785558    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 20:09:01.785690 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:01.785606    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 20:09:02.785518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:02.784903    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 20:09:02.785518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:02.784966    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 20:09:03.785895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:03.785859    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 20:09:03.786404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:03.785909    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 20:09:04.785635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:04.785598    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 20:09:04.785814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:04.785651    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 20:09:05.782291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:05.782244    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 01 20:09:05.782717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:05.782317    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 01 20:09:06.784879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:06.784849    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:09:06.785357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:06.785242    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:09:06.786049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:09:06.786030    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:09:21.627900 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:09:21.627859    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:09:51.627672 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:09:51.627628    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:10:21.627861 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:10:21.627823    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:10:28.536119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.536079    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 20:10:28.539530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.539507    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.592729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.592699    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\""
Nov 01 20:10:28.592956 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.592709    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 20:10:28.595551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.595523    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 20:10:28.607240 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.607195    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lhf\" (UniqueName: \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.607426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.607352    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.708842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.708802    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lhf\" (UniqueName: \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.709039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.708900    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.709602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.709581    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.765699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.765665    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lhf\" (UniqueName: \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\") pod \"payload-processing-55fbbbf9dc-fhnrf\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:28.853939 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:28.853858    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:29.224794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:29.224767    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 20:10:29.226893 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:10:29.226862    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc758dba3_c56f_4436_af91_e25f42d98603.slice/crio-07e1a4b3e70b09e12b7d1fe2c0e3f4c476addbb41408342e04c2fa914e9d1f7b WatchSource:0}: Error finding container 07e1a4b3e70b09e12b7d1fe2c0e3f4c476addbb41408342e04c2fa914e9d1f7b: Status 404 returned error can't find the container with id 07e1a4b3e70b09e12b7d1fe2c0e3f4c476addbb41408342e04c2fa914e9d1f7b
Nov 01 20:10:29.271554 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:29.271512    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" event={"ID":"c758dba3-c56f-4436-af91-e25f42d98603","Type":"ContainerStarted","Data":"07e1a4b3e70b09e12b7d1fe2c0e3f4c476addbb41408342e04c2fa914e9d1f7b"}
Nov 01 20:10:32.290023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:32.289978    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" event={"ID":"c758dba3-c56f-4436-af91-e25f42d98603","Type":"ContainerStarted","Data":"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"}
Nov 01 20:10:32.290410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:32.290050    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:32.403435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:32.403369    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" podStartSLOduration=1.8818403940000001 podStartE2EDuration="4.403311923s" totalImagesPullingTime="2.521471529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:32.402760796 +0000 UTC m=+485.787658387" watchObservedRunningTime="2026-11-01 20:10:32.403311923 +0000 UTC m=+485.788209508"
Nov 01 20:10:43.299435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:43.299348    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 20:10:51.628289 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:10:51.628253    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:10:58.640188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.640155    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"]
Nov 01 20:10:58.643717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.643697    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673486 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673452    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673622 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673589    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5fv\" (UniqueName: \"kubernetes.io/projected/0feecfe1-8475-436d-ad16-ac1f80c1208e-kube-api-access-jh5fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673621    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673650    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0feecfe1-8475-436d-ad16-ac1f80c1208e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673769 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673753    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.673867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.673850    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.702198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.702170    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"]
Nov 01 20:10:58.748614 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.748585    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 20:10:58.748735 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.748626    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wskl6\""
Nov 01 20:10:58.753053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.753036    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 20:10:58.753125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.753117    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 20:10:58.775019 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.774995    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5fv\" (UniqueName: \"kubernetes.io/projected/0feecfe1-8475-436d-ad16-ac1f80c1208e-kube-api-access-jh5fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775029    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775150    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0feecfe1-8475-436d-ad16-ac1f80c1208e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775203    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775378 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775354    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775399    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775551    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775710 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775691    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.775806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.775785    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.777279 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.777257    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0feecfe1-8475-436d-ad16-ac1f80c1208e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.777631 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.777615    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0feecfe1-8475-436d-ad16-ac1f80c1208e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.831323 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.831292    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5fv\" (UniqueName: \"kubernetes.io/projected/0feecfe1-8475-436d-ad16-ac1f80c1208e-kube-api-access-jh5fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc\" (UID: \"0feecfe1-8475-436d-ad16-ac1f80c1208e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:58.955857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:58.955791    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:10:59.326799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:59.326772    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"]
Nov 01 20:10:59.328943 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:10:59.328908    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feecfe1_8475_436d_ad16_ac1f80c1208e.slice/crio-b2809ee8c0f0a175ceac11882cf75c9cdd51f88a6e4c28d0f802b8dcea62d6d6 WatchSource:0}: Error finding container b2809ee8c0f0a175ceac11882cf75c9cdd51f88a6e4c28d0f802b8dcea62d6d6: Status 404 returned error can't find the container with id b2809ee8c0f0a175ceac11882cf75c9cdd51f88a6e4c28d0f802b8dcea62d6d6
Nov 01 20:10:59.426713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:10:59.426652    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc" event={"ID":"0feecfe1-8475-436d-ad16-ac1f80c1208e","Type":"ContainerStarted","Data":"b2809ee8c0f0a175ceac11882cf75c9cdd51f88a6e4c28d0f802b8dcea62d6d6"}
Nov 01 20:11:02.446001 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:02.445962    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc" event={"ID":"0feecfe1-8475-436d-ad16-ac1f80c1208e","Type":"ContainerStarted","Data":"d7af619b8c23f1fd2e8f1da557c126e86d4746a701e6fea41dfe986380df39f6"}
Nov 01 20:11:02.565151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:02.565056    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc" podStartSLOduration=2.319477267 podStartE2EDuration="4.565040921s" totalImagesPullingTime="2.245563654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:02.564145716 +0000 UTC m=+515.949043306" watchObservedRunningTime="2026-11-01 20:11:02.565040921 +0000 UTC m=+515.949938515"
Nov 01 20:11:02.956573 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:02.956544    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:11:08.956152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:08.956116    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:11:08.968290 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:08.968267    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:11:09.489815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:09.489785    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc"
Nov 01 20:11:18.988924 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:18.988887    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"]
Nov 01 20:11:18.992846 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:18.992823    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.067918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.067883    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gftg\" (UniqueName: \"kubernetes.io/projected/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-kube-api-access-9gftg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.068096 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.067957    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.068096 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.068076    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.068189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.068159    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.068229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.068209    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.068269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.068247    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.069899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.069878    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"]
Nov 01 20:11:19.096300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.096276    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 20:11:19.169051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169020    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169061    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169078    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169265    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169307    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169410    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169431    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gftg\" (UniqueName: \"kubernetes.io/projected/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-kube-api-access-9gftg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169518    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.169721 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.169695    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.171477 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.171455    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.171818 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.171798    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.228402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.228372    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gftg\" (UniqueName: \"kubernetes.io/projected/e0c79ef7-cb8d-454a-ad7a-04a878f241d4-kube-api-access-9gftg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n\" (UID: \"e0c79ef7-cb8d-454a-ad7a-04a878f241d4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.305455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.305377    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:19.676062 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:19.676034    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"]
Nov 01 20:11:19.677865 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:11:19.677843    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c79ef7_cb8d_454a_ad7a_04a878f241d4.slice/crio-7e8d4985dfc6de61a6c2e5a407bb9bb6ead05dd62ceceefa6116e21b5fef6711 WatchSource:0}: Error finding container 7e8d4985dfc6de61a6c2e5a407bb9bb6ead05dd62ceceefa6116e21b5fef6711: Status 404 returned error can't find the container with id 7e8d4985dfc6de61a6c2e5a407bb9bb6ead05dd62ceceefa6116e21b5fef6711
Nov 01 20:11:20.539778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:20.539697    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n" event={"ID":"e0c79ef7-cb8d-454a-ad7a-04a878f241d4","Type":"ContainerStarted","Data":"bd4f6ade956f8c21bb6c75fe968927bda036dc0530549403669a9b4b23d27b1a"}
Nov 01 20:11:20.539778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:20.539750    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n" event={"ID":"e0c79ef7-cb8d-454a-ad7a-04a878f241d4","Type":"ContainerStarted","Data":"7e8d4985dfc6de61a6c2e5a407bb9bb6ead05dd62ceceefa6116e21b5fef6711"}
Nov 01 20:11:20.655745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:20.655687    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n" podStartSLOduration=2.071438077 podStartE2EDuration="2.655672091s" totalImagesPullingTime="584.234014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:20.653779743 +0000 UTC m=+534.038677334" watchObservedRunningTime="2026-11-01 20:11:20.655672091 +0000 UTC m=+534.040569681"
Nov 01 20:11:21.306361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:21.306331    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:21.628658 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:11:21.628574    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:11:26.977697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:26.977655    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"]
Nov 01 20:11:26.982031 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:26.982005    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.085276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.085240    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 20:11:27.087895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.087865    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"]
Nov 01 20:11:27.147677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.147645    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa90c5-0265-43fa-b45d-6d52778e11df-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.147833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.147685    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcn5\" (UniqueName: \"kubernetes.io/projected/9caa90c5-0265-43fa-b45d-6d52778e11df-kube-api-access-kzcn5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.147833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.147778    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.147833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.147828    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.147970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.147949    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.148046 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.148028    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249116    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249200    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa90c5-0265-43fa-b45d-6d52778e11df-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249231    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcn5\" (UniqueName: \"kubernetes.io/projected/9caa90c5-0265-43fa-b45d-6d52778e11df-kube-api-access-kzcn5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249265    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249306    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249386    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249627    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249764    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.249783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.249775    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.251588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.251563    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9caa90c5-0265-43fa-b45d-6d52778e11df-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.251805 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.251773    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9caa90c5-0265-43fa-b45d-6d52778e11df-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.306281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.306244    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcn5\" (UniqueName: \"kubernetes.io/projected/9caa90c5-0265-43fa-b45d-6d52778e11df-kube-api-access-kzcn5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv\" (UID: \"9caa90c5-0265-43fa-b45d-6d52778e11df\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.601816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.601727    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:27.986283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:27.986259    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"]
Nov 01 20:11:27.988229 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:11:27.988202    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9caa90c5_0265_43fa_b45d_6d52778e11df.slice/crio-f95652c988f0e4b6153e5430e617d811c508a0e06fb456be3560d9ffba57faf9 WatchSource:0}: Error finding container f95652c988f0e4b6153e5430e617d811c508a0e06fb456be3560d9ffba57faf9: Status 404 returned error can't find the container with id f95652c988f0e4b6153e5430e617d811c508a0e06fb456be3560d9ffba57faf9
Nov 01 20:11:28.582931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:28.582904    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv" event={"ID":"9caa90c5-0265-43fa-b45d-6d52778e11df","Type":"ContainerStarted","Data":"f95652c988f0e4b6153e5430e617d811c508a0e06fb456be3560d9ffba57faf9"}
Nov 01 20:11:29.306198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.306163    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:29.318520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.318488    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:29.590111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.590009    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv" event={"ID":"9caa90c5-0265-43fa-b45d-6d52778e11df","Type":"ContainerStarted","Data":"0f16e1aabff26ac1c4470bb425428b750209ac77892e3c0d7edb429b3c8cd237"}
Nov 01 20:11:29.601337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.601309    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n"
Nov 01 20:11:29.602402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.602376    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:29.706423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:29.706361    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv" podStartSLOduration=3.120674417 podStartE2EDuration="3.706343462s" totalImagesPullingTime="585.669045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:29.705635187 +0000 UTC m=+543.090532779" watchObservedRunningTime="2026-11-01 20:11:29.706343462 +0000 UTC m=+543.091241055"
Nov 01 20:11:30.018513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.018467    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"]
Nov 01 20:11:30.022145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.022129    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.077274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.077230    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 20:11:30.079887 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.079855    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"]
Nov 01 20:11:30.184143 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184080    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.184394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184366    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb92p\" (UniqueName: \"kubernetes.io/projected/2171233f-acc1-4ada-84ad-bfdfa7d4c834-kube-api-access-jb92p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.184671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184652    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.184739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184713    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.184853 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184832    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.184916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.184867    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285642 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285554    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb92p\" (UniqueName: \"kubernetes.io/projected/2171233f-acc1-4ada-84ad-bfdfa7d4c834-kube-api-access-jb92p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285642 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285606    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285642 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285634    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285666    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285683    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.285913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.285724    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.286149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.286117    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.286218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.286141    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.286218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.286193    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.288113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.288095    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171233f-acc1-4ada-84ad-bfdfa7d4c834-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.288392 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.288373    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171233f-acc1-4ada-84ad-bfdfa7d4c834-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.342516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.342463    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb92p\" (UniqueName: \"kubernetes.io/projected/2171233f-acc1-4ada-84ad-bfdfa7d4c834-kube-api-access-jb92p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv\" (UID: \"2171233f-acc1-4ada-84ad-bfdfa7d4c834\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.343215 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.343194    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:30.716583 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:30.716557    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"]
Nov 01 20:11:30.719037 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:11:30.719012    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2171233f_acc1_4ada_84ad_bfdfa7d4c834.slice/crio-dbf5a57dbfbe82678db5c5477f33e581372efb0913dfacce3d0ea544f6e465d4 WatchSource:0}: Error finding container dbf5a57dbfbe82678db5c5477f33e581372efb0913dfacce3d0ea544f6e465d4: Status 404 returned error can't find the container with id dbf5a57dbfbe82678db5c5477f33e581372efb0913dfacce3d0ea544f6e465d4
Nov 01 20:11:31.604843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:31.604749    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" event={"ID":"2171233f-acc1-4ada-84ad-bfdfa7d4c834","Type":"ContainerStarted","Data":"30f917f7bd50e29246a7b828e2a0ac69df6e59c9e807bb11f6ef72312a3943c2"}
Nov 01 20:11:31.604843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:31.604793    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" event={"ID":"2171233f-acc1-4ada-84ad-bfdfa7d4c834","Type":"ContainerStarted","Data":"dbf5a57dbfbe82678db5c5477f33e581372efb0913dfacce3d0ea544f6e465d4"}
Nov 01 20:11:31.721194 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:31.721145    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podStartSLOduration=2.130574454 podStartE2EDuration="2.721132756s" totalImagesPullingTime="590.558302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:31.718876048 +0000 UTC m=+545.103773641" watchObservedRunningTime="2026-11-01 20:11:31.721132756 +0000 UTC m=+545.106030348"
Nov 01 20:11:32.343364 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:32.343334    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:37.602539 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:37.602506    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:37.614800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:37.614773    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:37.653186 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:37.653165    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv"
Nov 01 20:11:39.165954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.165919    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:11:39.169990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.169968    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.266991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.266958    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpqv\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267030    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267146    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267204    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267348    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267383    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267533    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267586    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.267633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.267615    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.368718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.368687    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:11:39.369133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369107    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369163    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369187    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369250    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369284    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369314    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369342    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpqv\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369484 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369406    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369484 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369438    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369651    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.369785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.369765    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.370084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.370055    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.370203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.370090    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.370203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.370195    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.372792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.372763    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.375193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.375167    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.426228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.426170    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.426508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.426476    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpqv\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2bhhr\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.483264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.483243    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:39.904146 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.904125    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:11:39.905658 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:11:39.905633    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc53d7d_8f90_4b8d_b9bd_c84bd3ae8f1e.slice/crio-ea6a745085e6195fce66c19ab3db164f1342adbc7b242dc50ff431621c8d9029 WatchSource:0}: Error finding container ea6a745085e6195fce66c19ab3db164f1342adbc7b242dc50ff431621c8d9029: Status 404 returned error can't find the container with id ea6a745085e6195fce66c19ab3db164f1342adbc7b242dc50ff431621c8d9029
Nov 01 20:11:39.907837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.907804    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:11:39.907940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.907868    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:11:39.907940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:39.907903    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:11:40.344227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.344182    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:40.361581 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.361547    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:40.660131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.660042    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" event={"ID":"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e","Type":"ContainerStarted","Data":"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"}
Nov 01 20:11:40.660131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.660110    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" event={"ID":"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e","Type":"ContainerStarted","Data":"ea6a745085e6195fce66c19ab3db164f1342adbc7b242dc50ff431621c8d9029"}
Nov 01 20:11:40.695862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.695831    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv"
Nov 01 20:11:40.779519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:40.778543    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podStartSLOduration=1.778523389 podStartE2EDuration="1.778523389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:40.777394105 +0000 UTC m=+554.162291733" watchObservedRunningTime="2026-11-01 20:11:40.778523389 +0000 UTC m=+554.163420981"
Nov 01 20:11:41.485712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:41.485673    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:41.487307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:41.487277    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:41.487438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:41.487369    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:42.483899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:42.483864    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:42.484210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:42.484180    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:43.491931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:43.491879    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:43.492448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:43.491948    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:44.485896 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:44.485852    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:44.486093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:44.485920    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:45.485883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:45.485846    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:45.486303 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:45.485924    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:46.484122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:46.484072    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:46.484341 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:46.484170    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:47.484080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:47.484029    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2bhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 20:11:47.484626 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:47.484108    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 20:11:49.131508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:49.131461    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:49.131968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:49.131946    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:49.132857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:11:49.132832    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:11:51.631600 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:11:51.631556    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:12:21.629472 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:21.629432    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:12:26.792211 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:26.792176    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:12:26.794161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:26.794134    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:12:36.944452 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:36.944416    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:12:36.948176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:36.948160    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.032224 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.032186    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxs9\" (UniqueName: \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.032554 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.032529    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.050967 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.050936    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d204f4\""
Nov 01 20:12:37.051150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.050982    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d204f4-dockercfg-k4p7m\""
Nov 01 20:12:37.053338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.053317    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:12:37.103179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.103150    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:12:37.106666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.106651    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.133365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.133338    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.133478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.133379    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxs9\" (UniqueName: \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.133478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.133412    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmf25\" (UniqueName: \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.133587 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.133538    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.134248 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.134229    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.163251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.163225    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:12:37.190081 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.190052    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxs9\" (UniqueName: \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\") pod \"payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.234330 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.234298    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.234527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.234341    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmf25\" (UniqueName: \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.235037 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.235015    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.261216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.261189    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:37.290693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.290663    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmf25\" (UniqueName: \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\") pod \"payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.420208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.420176    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.632479 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.632451    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:12:37.633633 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:37.633602    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7524e3e6_fe74_44ff_b9ae_e641fd63a54a.slice/crio-4ab40d6c84723e3bee0d936d51f738016448af610d26dedcd8ffbb7f6dbaf65d WatchSource:0}: Error finding container 4ab40d6c84723e3bee0d936d51f738016448af610d26dedcd8ffbb7f6dbaf65d: Status 404 returned error can't find the container with id 4ab40d6c84723e3bee0d936d51f738016448af610d26dedcd8ffbb7f6dbaf65d
Nov 01 20:12:37.789731 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.789705    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:12:37.791483 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:37.791442    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa85fee_1123_495e_98da_022d9b24b602.slice/crio-44abcc7707b1cadb6ad123e091cf1e24b249a1aa33d406a973420b7a6df2be11 WatchSource:0}: Error finding container 44abcc7707b1cadb6ad123e091cf1e24b249a1aa33d406a973420b7a6df2be11: Status 404 returned error can't find the container with id 44abcc7707b1cadb6ad123e091cf1e24b249a1aa33d406a973420b7a6df2be11
Nov 01 20:12:37.989053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.989016    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" event={"ID":"faa85fee-1123-495e-98da-022d9b24b602","Type":"ContainerStarted","Data":"f160b11353dcd81c3b6e9fdfbc0d39694005bbfdefc6f37838efb49322f69350"}
Nov 01 20:12:37.989585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.989082    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" event={"ID":"faa85fee-1123-495e-98da-022d9b24b602","Type":"ContainerStarted","Data":"44abcc7707b1cadb6ad123e091cf1e24b249a1aa33d406a973420b7a6df2be11"}
Nov 01 20:12:37.989585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.989232    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:37.990678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.990657    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" event={"ID":"7524e3e6-fe74-44ff-b9ae-e641fd63a54a","Type":"ContainerStarted","Data":"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"}
Nov 01 20:12:37.990743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.990696    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" event={"ID":"7524e3e6-fe74-44ff-b9ae-e641fd63a54a","Type":"ContainerStarted","Data":"4ab40d6c84723e3bee0d936d51f738016448af610d26dedcd8ffbb7f6dbaf65d"}
Nov 01 20:12:37.990743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:37.990716    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:38.110459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:38.110406    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" podStartSLOduration=1.110393222 podStartE2EDuration="1.110393222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:38.107585642 +0000 UTC m=+611.492483232" watchObservedRunningTime="2026-11-01 20:12:38.110393222 +0000 UTC m=+611.495290812"
Nov 01 20:12:38.251898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:38.251779    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" podStartSLOduration=2.251761229 podStartE2EDuration="2.251761229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:38.25125901 +0000 UTC m=+611.636156602" watchObservedRunningTime="2026-11-01 20:12:38.251761229 +0000 UTC m=+611.636658820"
Nov 01 20:12:39.637684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.637646    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:12:39.641562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.641538    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.658654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.658619    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.658819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.658670    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.658819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.658795    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.658819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.658817    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99ws\" (UniqueName: \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.745188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.745148    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:12:39.745374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.745163    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:12:39.745374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.745214    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-1b1134-serving-cert\""
Nov 01 20:12:39.745374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.745306    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 20:12:39.745374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.745326    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 20:12:39.747679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.747658    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:12:39.759941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.759907    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.760105 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:39.760040    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-serving-cert: secret "maas-api-e2e-worker-w3-1b1134-serving-cert" not found
Nov 01 20:12:39.760160 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:39.760108    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls podName:c4e04fb1-1e80-44d7-8b7e-576cf02f92c5 nodeName:}" failed. No retries permitted until 2026-11-01 20:12:40.260090691 +0000 UTC m=+613.644988261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls") pod "maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" (UID: "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5") : secret "maas-api-e2e-worker-w3-1b1134-serving-cert" not found
Nov 01 20:12:39.760203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.760189    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.760284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.760231    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.760381 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.760356    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99ws\" (UniqueName: \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.760901 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.760879    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.762692 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.762673    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:39.816997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:39.816963    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99ws\" (UniqueName: \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:40.265676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:40.265638    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:40.268109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:40.268084    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:40.555524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:40.555436    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:40.930657 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:40.930624    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:12:40.931810 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:40.931774    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e04fb1_1e80_44d7_8b7e_576cf02f92c5.slice/crio-d88468c46a5fb38db40d577e548ecd5229cede7335c09588482f749aa29ad4ec WatchSource:0}: Error finding container d88468c46a5fb38db40d577e548ecd5229cede7335c09588482f749aa29ad4ec: Status 404 returned error can't find the container with id d88468c46a5fb38db40d577e548ecd5229cede7335c09588482f749aa29ad4ec
Nov 01 20:12:40.933310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:40.933291    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:12:41.008612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.008579    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" event={"ID":"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5","Type":"ContainerStarted","Data":"d88468c46a5fb38db40d577e548ecd5229cede7335c09588482f749aa29ad4ec"}
Nov 01 20:12:41.685606 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.685575    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:12:41.690248 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.690232    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.742764 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.742730    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-381df3\""
Nov 01 20:12:41.744219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.744193    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-381df3-dockercfg-h2nxb\""
Nov 01 20:12:41.749157 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.749130    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:12:41.780915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.780884    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.781102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.780951    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvshb\" (UniqueName: \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.836266 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.836233    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:12:41.839902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.839878    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:41.882560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.882535    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:41.882696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.882603    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.882739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.882686    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslfb\" (UniqueName: \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:41.882739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.882733    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvshb\" (UniqueName: \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.883286 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.883267    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.896032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.896006    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:12:41.940694 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.940612    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvshb\" (UniqueName: \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\") pod \"payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:41.984054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.984008    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:41.984237 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.984141    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslfb\" (UniqueName: \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:41.984972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:41.984945    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:42.003810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:42.003783    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:42.041542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:42.041512    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslfb\" (UniqueName: \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\") pod \"payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:42.153780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:42.153747    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:42.390758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:42.390731    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:12:42.394577 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:42.394547    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cf60d6_d0e9_46da_82db_11997b8e0eea.slice/crio-dad0fd08178143558479c35292210d0ef434f3f5b908c187be3da257a8d81499 WatchSource:0}: Error finding container dad0fd08178143558479c35292210d0ef434f3f5b908c187be3da257a8d81499: Status 404 returned error can't find the container with id dad0fd08178143558479c35292210d0ef434f3f5b908c187be3da257a8d81499
Nov 01 20:12:42.549903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:42.549878    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:12:42.552287 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:42.552261    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054b71b5_54d7_4994_8e67_299a4b67f30c.slice/crio-773c4b31a05a7990e88ca496e1baafa1db0146ae58c3286bb117817edcffd46a WatchSource:0}: Error finding container 773c4b31a05a7990e88ca496e1baafa1db0146ae58c3286bb117817edcffd46a: Status 404 returned error can't find the container with id 773c4b31a05a7990e88ca496e1baafa1db0146ae58c3286bb117817edcffd46a
Nov 01 20:12:43.023942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.023902    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" event={"ID":"b8cf60d6-d0e9-46da-82db-11997b8e0eea","Type":"ContainerStarted","Data":"1b7354789ac8c35dc7ce8e46523d3d7483c80c858072677c022d4eb93c03cc86"}
Nov 01 20:12:43.024436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.023970    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" event={"ID":"b8cf60d6-d0e9-46da-82db-11997b8e0eea","Type":"ContainerStarted","Data":"dad0fd08178143558479c35292210d0ef434f3f5b908c187be3da257a8d81499"}
Nov 01 20:12:43.024436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.024007    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:43.026280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.026252    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" event={"ID":"054b71b5-54d7-4994-8e67-299a4b67f30c","Type":"ContainerStarted","Data":"99a47b5c3bada5bd35d8939aab5166a81198e8a2c2ad21c5a0870b47734eb0bb"}
Nov 01 20:12:43.026426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.026303    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" event={"ID":"054b71b5-54d7-4994-8e67-299a4b67f30c","Type":"ContainerStarted","Data":"773c4b31a05a7990e88ca496e1baafa1db0146ae58c3286bb117817edcffd46a"}
Nov 01 20:12:43.026426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.026374    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:43.137436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.137366    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" podStartSLOduration=2.137348929 podStartE2EDuration="2.137348929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:43.13664334 +0000 UTC m=+616.521540931" watchObservedRunningTime="2026-11-01 20:12:43.137348929 +0000 UTC m=+616.522246520"
Nov 01 20:12:43.252371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:43.252312    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" podStartSLOduration=2.252291715 podStartE2EDuration="2.252291715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:43.250809883 +0000 UTC m=+616.635707472" watchObservedRunningTime="2026-11-01 20:12:43.252291715 +0000 UTC m=+616.637189308"
Nov 01 20:12:44.106640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.106616    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:12:44.111555 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.111535    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.183072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.181706    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1b1134\""
Nov 01 20:12:44.185269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.185239    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-1b1134-dockercfg-gzgmm\""
Nov 01 20:12:44.187329 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.187301    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:12:44.219695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.219667    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7pdl\" (UniqueName: \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.219902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.219754    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.265901 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.265863    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:12:44.269799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.269778    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.320428 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.320399    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.320580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.320446    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.320580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.320559    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhth\" (UniqueName: \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.320657 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.320625    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7pdl\" (UniqueName: \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.321121 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.321104    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.327294 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.327234    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:12:44.377474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.377443    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7pdl\" (UniqueName: \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\") pod \"payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.421688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.421659    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.421830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.421730    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhth\" (UniqueName: \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.422370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.422352    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.435575 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.435544    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:44.478713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.478682    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhth\" (UniqueName: \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\") pod \"payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.582959 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.582880    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:44.808872 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.808841    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:12:44.809014 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:44.808992    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5d1eb2_eaae_4806_a983_5125ddc6b411.slice/crio-e442e9c01b6e18fc3e3cf8c19e87ae49c791b32a2d15936286dfae865427eeb6 WatchSource:0}: Error finding container e442e9c01b6e18fc3e3cf8c19e87ae49c791b32a2d15936286dfae865427eeb6: Status 404 returned error can't find the container with id e442e9c01b6e18fc3e3cf8c19e87ae49c791b32a2d15936286dfae865427eeb6
Nov 01 20:12:44.963450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:44.963412    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:12:44.963780 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:44.963756    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2a8edc_5560_4f44_88c9_570d0a8a5b95.slice/crio-0078228cb08a854edd8a2daf6b10dcc2c6b99a703a610689e09975885aa3fc68 WatchSource:0}: Error finding container 0078228cb08a854edd8a2daf6b10dcc2c6b99a703a610689e09975885aa3fc68: Status 404 returned error can't find the container with id 0078228cb08a854edd8a2daf6b10dcc2c6b99a703a610689e09975885aa3fc68
Nov 01 20:12:45.039016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.038984    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" event={"ID":"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5","Type":"ContainerStarted","Data":"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"}
Nov 01 20:12:45.039227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.039212    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:45.040746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.040724    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" event={"ID":"ca2a8edc-5560-4f44-88c9-570d0a8a5b95","Type":"ContainerStarted","Data":"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"}
Nov 01 20:12:45.040842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.040769    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:45.040842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.040791    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" event={"ID":"ca2a8edc-5560-4f44-88c9-570d0a8a5b95","Type":"ContainerStarted","Data":"0078228cb08a854edd8a2daf6b10dcc2c6b99a703a610689e09975885aa3fc68"}
Nov 01 20:12:45.042347 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.042323    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" event={"ID":"2b5d1eb2-eaae-4806-a983-5125ddc6b411","Type":"ContainerStarted","Data":"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"}
Nov 01 20:12:45.042484 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.042369    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" event={"ID":"2b5d1eb2-eaae-4806-a983-5125ddc6b411","Type":"ContainerStarted","Data":"e442e9c01b6e18fc3e3cf8c19e87ae49c791b32a2d15936286dfae865427eeb6"}
Nov 01 20:12:45.042484 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.042399    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:45.158007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.157915    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" podStartSLOduration=3.015407049 podStartE2EDuration="6.15790172s" totalImagesPullingTime="3.142494671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:45.155832005 +0000 UTC m=+618.540729596" watchObservedRunningTime="2026-11-01 20:12:45.15790172 +0000 UTC m=+618.542799343"
Nov 01 20:12:45.272955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.272856    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" podStartSLOduration=1.2728401200000001 podStartE2EDuration="1.27284012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:45.270620051 +0000 UTC m=+618.655517654" watchObservedRunningTime="2026-11-01 20:12:45.27284012 +0000 UTC m=+618.657737712"
Nov 01 20:12:45.385254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:45.385195    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" podStartSLOduration=1.385177707 podStartE2EDuration="1.385177707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:45.382567699 +0000 UTC m=+618.767465290" watchObservedRunningTime="2026-11-01 20:12:45.385177707 +0000 UTC m=+618.770075298"
Nov 01 20:12:48.999937 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:48.999909    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:12:49.000340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:48.999974    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:12:49.365051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.364963    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:12:49.369152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.369129    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.421982 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.421943    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-381df3-serving-cert\""
Nov 01 20:12:49.425367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.425338    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:12:49.474337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.474309    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.474518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.474346    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.474518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.474376    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv4s\" (UniqueName: \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.474644 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.474554    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.575406 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.575367    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.575595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.575424    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv4s\" (UniqueName: \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.575595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.575461    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.575595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.575563    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.575717 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:49.575671    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-serving-cert: secret "maas-api-e2e-praxis-a-381df3-serving-cert" not found
Nov 01 20:12:49.575757 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:49.575743    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls podName:71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734 nodeName:}" failed. No retries permitted until 2026-11-01 20:12:50.075722242 +0000 UTC m=+623.460619815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls") pod "maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" (UID: "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734") : secret "maas-api-e2e-praxis-a-381df3-serving-cert" not found
Nov 01 20:12:49.576266 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.576244    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.577859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.577840    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:49.632783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:49.632700    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv4s\" (UniqueName: \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:50.081516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:50.081462    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:50.083877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:50.083856    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") pod \"maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:50.283184 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:50.283146    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:50.660108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:50.660075    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:12:50.660626 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:12:50.660599    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e1b9e5_6f1e_44f4_b8a2_39eb5d9c6734.slice/crio-c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc WatchSource:0}: Error finding container c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc: Status 404 returned error can't find the container with id c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc
Nov 01 20:12:51.054360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:51.054330    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:12:51.079770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:51.079724    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" event={"ID":"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734","Type":"ContainerStarted","Data":"c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc"}
Nov 01 20:12:51.628746 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:12:51.628709    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:12:52.087168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:52.087126    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" event={"ID":"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734","Type":"ContainerStarted","Data":"aeaec6d54eafdfc76eb09212eccba3ea0161a322860fc0fc65cdf845fb86d4b7"}
Nov 01 20:12:52.087345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:52.087326    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:12:52.203359 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:52.203294    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" podStartSLOduration=2.426523091 podStartE2EDuration="3.2032757s" totalImagesPullingTime="776.752609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:52.202690667 +0000 UTC m=+625.587588260" watchObservedRunningTime="2026-11-01 20:12:52.2032757 +0000 UTC m=+625.588173298"
Nov 01 20:12:54.035359 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:54.035334    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:12:54.035876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:54.035400    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:12:56.052281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:56.052247    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:12:56.052768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:56.052322    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:12:58.101678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:12:58.101644    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:13:09.913383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:09.913348    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:13:09.918635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:09.918608    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.106175 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.106131    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.106472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.106444    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.107187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.106650    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.107410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.107386    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5h8b\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.107593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.107573    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.108151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.108129    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.108366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.108326    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.109010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.108988    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.109206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.109185    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.150777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.150735    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:13:10.210576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.210460    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.210836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.210809    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.210978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.210961    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211084    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5h8b\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211241 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211225    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211342    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211481 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211466    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211647 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211618    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.211821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.211802    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.212447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.212423    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.212888 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.212868    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.213837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.213814    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.215125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.215104    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.215885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.215829    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.218103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.218080    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.222048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.222026    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.282966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.282933    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.297393 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.297311    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5h8b\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-drkzb\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.535771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.535735    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:10.973250 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.973208    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:13:10.979838 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:10.979804    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551fc23f_2bca_44c2_9b06_2fd9cc626119.slice/crio-4af4f26745dbc4902afc93c05a405a8966b5d9c09686e912695708594b798237 WatchSource:0}: Error finding container 4af4f26745dbc4902afc93c05a405a8966b5d9c09686e912695708594b798237: Status 404 returned error can't find the container with id 4af4f26745dbc4902afc93c05a405a8966b5d9c09686e912695708594b798237
Nov 01 20:13:10.982707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.982668    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:10.982820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.982760    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:10.982820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:10.982800    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:11.215698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:11.215653    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" event={"ID":"551fc23f-2bca-44c2-9b06-2fd9cc626119","Type":"ContainerStarted","Data":"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"}
Nov 01 20:13:11.215884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:11.215727    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" event={"ID":"551fc23f-2bca-44c2-9b06-2fd9cc626119","Type":"ContainerStarted","Data":"4af4f26745dbc4902afc93c05a405a8966b5d9c09686e912695708594b798237"}
Nov 01 20:13:11.333233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:11.333131    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podStartSLOduration=2.333117091 podStartE2EDuration="2.333117091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:11.331332565 +0000 UTC m=+644.716230158" watchObservedRunningTime="2026-11-01 20:13:11.333117091 +0000 UTC m=+644.718014819"
Nov 01 20:13:11.536055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:11.536018    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:12.538171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:12.538133    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:12.538590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:12.538187    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:13.536790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:13.536755    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:13.536968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:13.536819    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:14.536509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:14.536459    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:14.536903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:14.536541    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:15.544102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:15.543623    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:15.544102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:15.543702    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:16.536806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:16.536771    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:16.536995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:16.536841    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:17.537020 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:17.536977    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:17.537417 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:17.537046    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:18.537026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:18.536981    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-drkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 20:13:18.537417 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:18.537049    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 20:13:20.440355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:20.440324    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:20.440829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:20.440778    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:20.441454 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:20.441434    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:13:21.573715 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.573681    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:13:21.581964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.581936    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.633983 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.633955    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cb8d4e-openshift-default-dockercfg-6xt6g\""
Nov 01 20:13:21.660357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660320    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660363    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56jx\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660411    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660429    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660446    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660479    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660527    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660591    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.660666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.660613    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.678915 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:13:21.678878    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:13:21.687275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.687237    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:13:21.762212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762167    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762233    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762298    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762333    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56jx\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762404    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762430    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762638    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762644    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762672    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762720    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762755    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762794    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.762948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.762919    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.763441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.763416    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.764668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.764645    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.765157 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.765133    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.819719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.819689    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.819918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.819899    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56jx\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\") pod \"e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") " pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:21.897101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:21.897026    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:22.377380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:22.377353    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:13:22.417472 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:22.417434    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb540aa58_cca9_46fe_81b9_8564f2673e54.slice/crio-5a70bdf7dc101f8d36edc70c7f544b69a17df0ef3b9623b8d5eb90c6332e2640 WatchSource:0}: Error finding container 5a70bdf7dc101f8d36edc70c7f544b69a17df0ef3b9623b8d5eb90c6332e2640: Status 404 returned error can't find the container with id 5a70bdf7dc101f8d36edc70c7f544b69a17df0ef3b9623b8d5eb90c6332e2640
Nov 01 20:13:22.419640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:22.419604    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:22.419737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:22.419674    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:22.419737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:22.419702    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:13:23.286509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:23.286464    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerStarted","Data":"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"}
Nov 01 20:13:23.286928 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:23.286539    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerStarted","Data":"5a70bdf7dc101f8d36edc70c7f544b69a17df0ef3b9623b8d5eb90c6332e2640"}
Nov 01 20:13:23.500771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:23.500723    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podStartSLOduration=2.500708449 podStartE2EDuration="2.500708449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:23.450610686 +0000 UTC m=+656.835508278" watchObservedRunningTime="2026-11-01 20:13:23.500708449 +0000 UTC m=+656.885606040"
Nov 01 20:13:23.897603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:23.897566    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:23.903226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:23.903202    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:24.292406 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:24.292367    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:24.293330 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:24.293310    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:13:29.847762 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.847724    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:13:29.852852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.852829    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:29.913519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.913461    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:13:29.944444 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.944414    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cb8d4e\""
Nov 01 20:13:29.952891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.952808    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fr94\" (UniqueName: \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:29.953054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.953039    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:29.953094 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.953056    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cb8d4e-dockercfg-m65tn\""
Nov 01 20:13:29.980774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.980744    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:13:29.984736 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:29.984713    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.044070 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.044038    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:13:30.054178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.054135    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqp2\" (UniqueName: \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.054329 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.054192    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:30.054329 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.054280    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.054414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.054394    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fr94\" (UniqueName: \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:30.055084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.055068    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:30.111527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.111434    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fr94\" (UniqueName: \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\") pod \"payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:30.156019 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.155984    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqp2\" (UniqueName: \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.156192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.156029    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.156775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.156755    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.165646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.165629    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:30.216646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.215864    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqp2\" (UniqueName: \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\") pod \"payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.297293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.297264    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:30.538029 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.538004    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:13:30.539183 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:30.539152    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79439885_c38c_4154_8665_d2f28b94ec32.slice/crio-711b289f41db57dece8c2535f0a2ec037820163f0751486d4d2efbc91a2695e0 WatchSource:0}: Error finding container 711b289f41db57dece8c2535f0a2ec037820163f0751486d4d2efbc91a2695e0: Status 404 returned error can't find the container with id 711b289f41db57dece8c2535f0a2ec037820163f0751486d4d2efbc91a2695e0
Nov 01 20:13:30.676024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:30.675988    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:13:30.676179 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:30.676150    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85990ac_6933_4a19_8081_a1b45896012c.slice/crio-0fc434dcaa27362b945a4cf846587a2ee1e08a45118a79eac1232d4e6a644f12 WatchSource:0}: Error finding container 0fc434dcaa27362b945a4cf846587a2ee1e08a45118a79eac1232d4e6a644f12: Status 404 returned error can't find the container with id 0fc434dcaa27362b945a4cf846587a2ee1e08a45118a79eac1232d4e6a644f12
Nov 01 20:13:31.335388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.335351    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" event={"ID":"79439885-c38c-4154-8665-d2f28b94ec32","Type":"ContainerStarted","Data":"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"}
Nov 01 20:13:31.335907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.335407    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" event={"ID":"79439885-c38c-4154-8665-d2f28b94ec32","Type":"ContainerStarted","Data":"711b289f41db57dece8c2535f0a2ec037820163f0751486d4d2efbc91a2695e0"}
Nov 01 20:13:31.335907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.335489    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:31.337145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.337120    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" event={"ID":"a85990ac-6933-4a19-8081-a1b45896012c","Type":"ContainerStarted","Data":"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"}
Nov 01 20:13:31.337285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.337161    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" event={"ID":"a85990ac-6933-4a19-8081-a1b45896012c","Type":"ContainerStarted","Data":"0fc434dcaa27362b945a4cf846587a2ee1e08a45118a79eac1232d4e6a644f12"}
Nov 01 20:13:31.337285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.337204    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:31.449776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.449716    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" podStartSLOduration=2.4497015380000002 podStartE2EDuration="2.449701538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:31.447648667 +0000 UTC m=+664.832546259" watchObservedRunningTime="2026-11-01 20:13:31.449701538 +0000 UTC m=+664.834599128"
Nov 01 20:13:31.563630 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:31.563117    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" podStartSLOduration=2.5630954900000003 podStartE2EDuration="2.56309549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:31.561285608 +0000 UTC m=+664.946183224" watchObservedRunningTime="2026-11-01 20:13:31.56309549 +0000 UTC m=+664.947993099"
Nov 01 20:13:34.860512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:34.860471    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:13:34.864562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:34.864543    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:34.917024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:34.916992    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-df41b8\""
Nov 01 20:13:34.966507 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:34.966473    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-df41b8-dockercfg-mt969\""
Nov 01 20:13:34.969243 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:34.969223    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:13:35.020654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.020627    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf5tp\" (UniqueName: \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.020810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.020692    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.122291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.122214    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf5tp\" (UniqueName: \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.122291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.122283    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.123035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.123014    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.178996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.178958    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf5tp\" (UniqueName: \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\") pod \"payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.479508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.479462    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:35.866191 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:35.866151    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:13:35.867318 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:35.867288    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb616f7de_b7e5_4f09_ab82_695dbb43e1d3.slice/crio-45fafc8d741e35bd179a2b263aa26d066da682de8a4e10b6e91c584f9a08b664 WatchSource:0}: Error finding container 45fafc8d741e35bd179a2b263aa26d066da682de8a4e10b6e91c584f9a08b664: Status 404 returned error can't find the container with id 45fafc8d741e35bd179a2b263aa26d066da682de8a4e10b6e91c584f9a08b664
Nov 01 20:13:36.374388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:36.374342    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" event={"ID":"b616f7de-b7e5-4f09-ab82-695dbb43e1d3","Type":"ContainerStarted","Data":"bae77dc350202b5eaf880eb534d7b846f2fa211bdc4fd2d6f041fc34847e493b"}
Nov 01 20:13:36.374590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:36.374413    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" event={"ID":"b616f7de-b7e5-4f09-ab82-695dbb43e1d3","Type":"ContainerStarted","Data":"45fafc8d741e35bd179a2b263aa26d066da682de8a4e10b6e91c584f9a08b664"}
Nov 01 20:13:36.374975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:36.374923    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:36.488636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:36.488561    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" podStartSLOduration=2.488543388 podStartE2EDuration="2.488543388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:36.487408242 +0000 UTC m=+669.872305845" watchObservedRunningTime="2026-11-01 20:13:36.488543388 +0000 UTC m=+669.873440983"
Nov 01 20:13:42.347342 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:42.347128    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:13:42.347342 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:42.347219    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:13:46.752845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.752804    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:13:46.757061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.757035    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.815105 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.815063    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cb8d4e-serving-cert\""
Nov 01 20:13:46.818440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.818405    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:13:46.846750 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.846702    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57cx\" (UniqueName: \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.846941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.846841    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.846941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.846891    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.847065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.846995    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.947980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.947933    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.948198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.947995    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.948198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.948040    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.948198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.948084    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57cx\" (UniqueName: \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.948198 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:13:46.948111    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-serving-cert: secret "maas-api-e2e-praxis-b-cb8d4e-serving-cert" not found
Nov 01 20:13:46.948198 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:13:46.948199    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls podName:1eaad4c6-ef99-4420-bda5-c428e7e61c48 nodeName:}" failed. No retries permitted until 2026-11-01 20:13:47.448176342 +0000 UTC m=+680.833073928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls") pod "maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" (UID: "1eaad4c6-ef99-4420-bda5-c428e7e61c48") : secret "maas-api-e2e-praxis-b-cb8d4e-serving-cert" not found
Nov 01 20:13:46.948752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.948732    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:46.950468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:46.950441    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:47.016957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:47.016866    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57cx\" (UniqueName: \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:47.383056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:47.382969    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:13:47.454981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:47.454941    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:47.457398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:47.457373    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:47.670127 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:47.670042    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:48.099023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:48.098992    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:13:48.102341 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:13:48.102315    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaad4c6_ef99_4420_bda5_c428e7e61c48.slice/crio-bc4b32be7635fb7ceb39f2137db2ea739aad5080cd184a6586dfad187a72dfed WatchSource:0}: Error finding container bc4b32be7635fb7ceb39f2137db2ea739aad5080cd184a6586dfad187a72dfed: Status 404 returned error can't find the container with id bc4b32be7635fb7ceb39f2137db2ea739aad5080cd184a6586dfad187a72dfed
Nov 01 20:13:48.453535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:48.453422    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" event={"ID":"1eaad4c6-ef99-4420-bda5-c428e7e61c48","Type":"ContainerStarted","Data":"bc4b32be7635fb7ceb39f2137db2ea739aad5080cd184a6586dfad187a72dfed"}
Nov 01 20:13:49.462174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:49.462075    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" event={"ID":"1eaad4c6-ef99-4420-bda5-c428e7e61c48","Type":"ContainerStarted","Data":"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"}
Nov 01 20:13:49.462174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:49.462139    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:13:49.580261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:49.576103    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" podStartSLOduration=2.657488824 podStartE2EDuration="3.576086568s" totalImagesPullingTime="918.597744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:49.575378166 +0000 UTC m=+682.960275758" watchObservedRunningTime="2026-11-01 20:13:49.576086568 +0000 UTC m=+682.960984158"
Nov 01 20:13:51.629415 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:13:51.629379    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:13:55.473237 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:13:55.473200    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:14:09.294404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.294365    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:09.294923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.294443    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:09.589570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.589437    5066 generic.go:356] "Generic (PLEG): container finished" podID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerID="59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4" exitCode=0
Nov 01 20:14:09.589570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.589526    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerDied","Data":"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"}
Nov 01 20:14:09.589989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.589972    5066 scope.go:122] "RemoveContainer" containerID="59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"
Nov 01 20:14:09.591343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.591316    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:09.591443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.591380    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:09.591443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:09.591410    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:10.598030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:10.597993    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerStarted","Data":"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"}
Nov 01 20:14:10.897699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:10.897604    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:14:10.897884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:10.897813    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:10.897884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:10.897866    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:11.897941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:11.897908    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:11.898343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:11.897960    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:12.898337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:12.898303    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:12.898856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:12.898364    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:13.897889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:13.897849    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:13.898084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:13.897902    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:14.897696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:14.897655    5066 patch_prober.go:29] interesting pod/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 20:14:14.898101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:14.897715    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 20:14:14.986677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:14.986639    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:14:14.991739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:14.991714    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.044187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.044152    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1b1134-openshift-default-dockercfg-pvzt4\""
Nov 01 20:14:15.056969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.056934    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.057255 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.057227    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.057405 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.057384    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjwjb\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.057625 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.057603    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.057760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.057743    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.057906 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.057887    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.058109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.058090    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.058309 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.058290    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.058478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.058458    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.096189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.096152    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:14:15.185877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.185781    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.186597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.186571    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.186772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.186755    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjwjb\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.199727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.186948    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.199727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.199467    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.199727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.189051    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.199727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.199378    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.199727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.189811    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200232    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200280    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200335    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200394    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200546    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200709 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200688    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200733    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.200818 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.200763    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.253296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.253247    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjwjb\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.281481 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.281446    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.306249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.306215    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:15.775533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.773383    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:14:15.780049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.779994    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:15.780165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.780140    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:15.780247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.780226    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:14:15.902409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.902372    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:14:15.902971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.902952    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:14:15.903744 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:15.903723    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:14:16.636368 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:16.636328    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" event={"ID":"2462138c-3486-4f81-a5b3-46b2860b3d4d","Type":"ContainerStarted","Data":"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"}
Nov 01 20:14:16.636561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:16.636397    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" event={"ID":"2462138c-3486-4f81-a5b3-46b2860b3d4d","Type":"ContainerStarted","Data":"265a9ba91e64b60f0390a47c7bc6aef594d0222e7675a4fd7cf6fbdae1946cd3"}
Nov 01 20:14:16.753115 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:16.753061    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podStartSLOduration=2.753045903 podStartE2EDuration="2.753045903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:16.752059116 +0000 UTC m=+710.136956708" watchObservedRunningTime="2026-11-01 20:14:16.753045903 +0000 UTC m=+710.137943493"
Nov 01 20:14:17.306843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:17.306805    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:17.308672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:17.308645    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:17.308780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:17.308694    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:18.306949 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:18.306919    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:18.307359 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:18.306987    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:19.306771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:19.306733    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:19.306985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:19.306787    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:20.307199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:20.307164    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:20.307835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:20.307802    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:21.307683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:21.307643    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:21.308317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:21.308285    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:21.629169 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:14:21.629089    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:14:22.306959 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:22.306914    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 20:14:22.307155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:22.306999    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 20:14:23.711609 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:23.711577    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:23.712157 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:23.712127    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:23.712989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:23.712965    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:14:45.188560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.188463    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:14:45.192458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.192434    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286320    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286373    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkr4s\" (UniqueName: \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286574    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286684    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286735    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.286830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.286818    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.297316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.297284    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-381df3\"/\"kube-root-ca.crt\""
Nov 01 20:14:45.297316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.297306    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-381df3\"/\"praxis-route-381df3-kserve-self-signed-certs\""
Nov 01 20:14:45.297528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.297342    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-381df3\"/\"openshift-service-ca.crt\""
Nov 01 20:14:45.297805 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.297531    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-381df3\"/\"default-dockercfg-2m8t9\""
Nov 01 20:14:45.300267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.300244    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:14:45.387889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.387850    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.387889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.387900    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkr4s\" (UniqueName: \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.388487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.387991    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.388487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.388064    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.388487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.388089    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.388487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.388151    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.388900 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.388871    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.389032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.388965    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.389146 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.389120    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.390825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.390792    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.391107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.391083    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.445371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.445290    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkr4s\" (UniqueName: \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\") pod \"praxis-route-381df3-kserve-67cd57b67f-c5f7h\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") " pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.505696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.505657    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:45.911524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:45.911482    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:14:45.913204 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:14:45.913173    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefeb7e65_9f4a_424c_bedd_1943c98f3aaa.slice/crio-94a101c7827ccb6bbcbc063254cbbe7a59a2f6486ec285c4cd8c74aadfe19b62 WatchSource:0}: Error finding container 94a101c7827ccb6bbcbc063254cbbe7a59a2f6486ec285c4cd8c74aadfe19b62: Status 404 returned error can't find the container with id 94a101c7827ccb6bbcbc063254cbbe7a59a2f6486ec285c4cd8c74aadfe19b62
Nov 01 20:14:46.846565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:46.846522    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" event={"ID":"efeb7e65-9f4a-424c-bedd-1943c98f3aaa","Type":"ContainerStarted","Data":"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"}
Nov 01 20:14:46.847038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:46.846575    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" event={"ID":"efeb7e65-9f4a-424c-bedd-1943c98f3aaa","Type":"ContainerStarted","Data":"94a101c7827ccb6bbcbc063254cbbe7a59a2f6486ec285c4cd8c74aadfe19b62"}
Nov 01 20:14:46.960470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:46.960415    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" podStartSLOduration=1.960399606 podStartE2EDuration="1.960399606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:46.959358517 +0000 UTC m=+740.344256109" watchObservedRunningTime="2026-11-01 20:14:46.960399606 +0000 UTC m=+740.345297198"
Nov 01 20:14:47.506621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:47.506584    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:51.630622 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:14:51.630574    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:14:55.506271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:55.506233    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:55.519074 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:55.519049    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:55.912292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:55.912212    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:14:59.956011 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:59.955952    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:14:59.961618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:14:59.961595    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.013640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.013596    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\""
Nov 01 20:15:00.051310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051268    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051310    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051466    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbvm\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051554    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051594    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051757    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051844 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051812    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051842    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.051890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.051877    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.067465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.067396    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:15:00.153551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153478    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153576    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153600    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153624    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153673    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153699    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153789    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbvm\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153828    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.153989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.153870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.154161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.154041    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.154161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.154055    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.154271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.154162    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.154333 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.154302    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.154487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.154462    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.156080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.156058    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.156183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.156137    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.196104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.196061    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:00.200433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.200408    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.210798 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.210766    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.211089 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.211066    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbvm\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.255723 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.255510    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:00.276701 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.276668    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:00.341281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.341252    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\""
Nov 01 20:15:00.356821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.356735    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.356821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.356781    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgnf\" (UniqueName: \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.457633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.457591    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.457633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.457632    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgnf\" (UniqueName: \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.458251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.458228    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.513746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.513711    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgnf\" (UniqueName: \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\") pod \"maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.516545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.516515    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:00.707833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.707798    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:15:00.709063 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:15:00.709033    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af0d1d_9857_4cfe_b4f0_04c03f36ee56.slice/crio-75c62a5fc62a27c066c1f2bd18dc5677237579c91ace8efb56e61476a28e6f57 WatchSource:0}: Error finding container 75c62a5fc62a27c066c1f2bd18dc5677237579c91ace8efb56e61476a28e6f57: Status 404 returned error can't find the container with id 75c62a5fc62a27c066c1f2bd18dc5677237579c91ace8efb56e61476a28e6f57
Nov 01 20:15:00.711390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.711358    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:15:00.711478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.711422    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:15:00.711478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.711453    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:15:00.891149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.891122    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:00.892261 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:15:00.892239    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9193dfec_ed7e_4686_a413_76ddf6e0a1e7.slice/crio-21bdd4729c2dbb38cb4f0355ef86733ce9b8b83cd2cd93cb2eb79dd0c2fbdb0b WatchSource:0}: Error finding container 21bdd4729c2dbb38cb4f0355ef86733ce9b8b83cd2cd93cb2eb79dd0c2fbdb0b: Status 404 returned error can't find the container with id 21bdd4729c2dbb38cb4f0355ef86733ce9b8b83cd2cd93cb2eb79dd0c2fbdb0b
Nov 01 20:15:00.934735 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.934697    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerStarted","Data":"21bdd4729c2dbb38cb4f0355ef86733ce9b8b83cd2cd93cb2eb79dd0c2fbdb0b"}
Nov 01 20:15:00.936278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.936252    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" event={"ID":"57af0d1d-9857-4cfe-b4f0-04c03f36ee56","Type":"ContainerStarted","Data":"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"}
Nov 01 20:15:00.936425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:00.936294    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" event={"ID":"57af0d1d-9857-4cfe-b4f0-04c03f36ee56","Type":"ContainerStarted","Data":"75c62a5fc62a27c066c1f2bd18dc5677237579c91ace8efb56e61476a28e6f57"}
Nov 01 20:15:01.058027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:01.057913    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podStartSLOduration=2.057891848 podStartE2EDuration="2.057891848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:01.054220018 +0000 UTC m=+754.439117611" watchObservedRunningTime="2026-11-01 20:15:01.057891848 +0000 UTC m=+754.442789439"
Nov 01 20:15:01.283516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:01.280879    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:01.288730 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:01.288679    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:01.289027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:01.288973    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:02.288025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.287951    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:02.288025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.288021    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:02.929518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.921338    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:15:02.929518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.927390    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:02.931562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.931523    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:02.931562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.931572    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzrf\" (UniqueName: \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:02.942617 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.942282    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:15:02.998914 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.998860    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ad7edb\""
Nov 01 20:15:02.999211 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:02.999181    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ad7edb-dockercfg-jfvc8\""
Nov 01 20:15:03.032831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.032552    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:03.032831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.032595    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzrf\" (UniqueName: \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:03.033346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.033323    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:03.098201 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.098136    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzrf\" (UniqueName: \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\") pod \"payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:03.245781 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.245740    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:03.283088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.282928    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:03.307751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.283725    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:03.707564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.704373    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:15:03.707564 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:15:03.707403    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode559ba6a_9ff9_4219_997d_3c16639eea51.slice/crio-212b252fa2ee7438babc918f9306c2aa254f089161a5f4a4959ec1773e69000a WatchSource:0}: Error finding container 212b252fa2ee7438babc918f9306c2aa254f089161a5f4a4959ec1773e69000a: Status 404 returned error can't find the container with id 212b252fa2ee7438babc918f9306c2aa254f089161a5f4a4959ec1773e69000a
Nov 01 20:15:03.958671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.958514    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" event={"ID":"e559ba6a-9ff9-4219-997d-3c16639eea51","Type":"ContainerStarted","Data":"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"}
Nov 01 20:15:03.958671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.958581    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" event={"ID":"e559ba6a-9ff9-4219-997d-3c16639eea51","Type":"ContainerStarted","Data":"212b252fa2ee7438babc918f9306c2aa254f089161a5f4a4959ec1773e69000a"}
Nov 01 20:15:03.958671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:03.958629    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:04.280427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:04.279959    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:04.280427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:04.280033    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:05.286682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:05.286210    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:05.286682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:05.286271    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:06.280951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:06.280911    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:06.281255 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:06.281226    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:07.000940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.000904    5066 generic.go:356] "Generic (PLEG): container finished" podID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerID="d491d4c336c4abebcf1859eba60b89fdefd074e17c633cc778fa8ac2692d08b4" exitCode=60
Nov 01 20:15:07.001443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.000975    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerDied","Data":"d491d4c336c4abebcf1859eba60b89fdefd074e17c633cc778fa8ac2692d08b4"}
Nov 01 20:15:07.001443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.001330    5066 scope.go:122] "RemoveContainer" containerID="d491d4c336c4abebcf1859eba60b89fdefd074e17c633cc778fa8ac2692d08b4"
Nov 01 20:15:07.115804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.115738    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" podStartSLOduration=5.115719245 podStartE2EDuration="5.115719245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:04.087886187 +0000 UTC m=+757.472783779" watchObservedRunningTime="2026-11-01 20:15:07.115719245 +0000 UTC m=+760.500616838"
Nov 01 20:15:07.277835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.277798    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 20:15:07.277963 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:07.277861    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 20:15:08.009615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.009577    5066 generic.go:356] "Generic (PLEG): container finished" podID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerID="7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013" exitCode=60
Nov 01 20:15:08.010050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.009649    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerDied","Data":"7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013"}
Nov 01 20:15:08.010050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.009708    5066 scope.go:122] "RemoveContainer" containerID="d491d4c336c4abebcf1859eba60b89fdefd074e17c633cc778fa8ac2692d08b4"
Nov 01 20:15:08.010050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.009977    5066 scope.go:122] "RemoveContainer" containerID="7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013"
Nov 01 20:15:08.010282 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:15:08.010260    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t_odh-ai-gateway-infra(9193dfec-ed7e-4686-a413-76ddf6e0a1e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7"
Nov 01 20:15:08.281862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.281788    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:08.282090 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.282071    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:08.282905 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:08.282890    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:15:09.018285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:09.018251    5066 scope.go:122] "RemoveContainer" containerID="7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013"
Nov 01 20:15:09.018713 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:15:09.018602    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t_odh-ai-gateway-infra(9193dfec-ed7e-4686-a413-76ddf6e0a1e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7"
Nov 01 20:15:14.989205 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:14.989173    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:15:18.724017 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:18.723980    5066 scope.go:122] "RemoveContainer" containerID="7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013"
Nov 01 20:15:19.085599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:19.085570    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerStarted","Data":"b917f9ffdab63d77c4507529407a47c097ca8e5f0fb744622de23341ebdb9b99"}
Nov 01 20:15:19.198512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:19.198433    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" podStartSLOduration=14.420781454 podStartE2EDuration="19.198415538s" totalImagesPullingTime="4.777634084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:19.197407393 +0000 UTC m=+772.582304984" watchObservedRunningTime="2026-11-01 20:15:19.198415538 +0000 UTC m=+772.583313128"
Nov 01 20:15:20.094793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:20.094757    5066 generic.go:356] "Generic (PLEG): container finished" podID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerID="b917f9ffdab63d77c4507529407a47c097ca8e5f0fb744622de23341ebdb9b99" exitCode=60
Nov 01 20:15:20.095296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:20.094826    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerDied","Data":"b917f9ffdab63d77c4507529407a47c097ca8e5f0fb744622de23341ebdb9b99"}
Nov 01 20:15:20.095296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:20.094884    5066 scope.go:122] "RemoveContainer" containerID="7d6abfba1b4c634d19bacbd62c67ab34e1a42f1d57b7709126a2696f812c7013"
Nov 01 20:15:20.095296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:20.095290    5066 scope.go:122] "RemoveContainer" containerID="b917f9ffdab63d77c4507529407a47c097ca8e5f0fb744622de23341ebdb9b99"
Nov 01 20:15:20.095660 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:15:20.095638    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t_odh-ai-gateway-infra(9193dfec-ed7e-4686-a413-76ddf6e0a1e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7"
Nov 01 20:15:20.209232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:20.209195    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:21.258698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.258670    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:21.390967 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.390876    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\") pod \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") "
Nov 01 20:15:21.391119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.391049    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\" (UniqueName: \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\") pod \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\" (UID: \"9193dfec-ed7e-4686-a413-76ddf6e0a1e7\") "
Nov 01 20:15:21.391314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.391287    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca" pod "9193dfec-ed7e-4686-a413-76ddf6e0a1e7" (UID: "9193dfec-ed7e-4686-a413-76ddf6e0a1e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:21.391572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.391546    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:15:21.393454 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.393424    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf" pod "9193dfec-ed7e-4686-a413-76ddf6e0a1e7" (UID: "9193dfec-ed7e-4686-a413-76ddf6e0a1e7"). InnerVolumeSpecName "kube-api-access-qzgnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:21.492351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:21.492317    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgnf\" (UniqueName: \"kubernetes.io/projected/9193dfec-ed7e-4686-a413-76ddf6e0a1e7-kube-api-access-qzgnf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:15:21.629504 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:15:21.629454    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:15:22.111011 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.110980    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"
Nov 01 20:15:22.111197 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.111023    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t" event={"ID":"9193dfec-ed7e-4686-a413-76ddf6e0a1e7","Type":"ContainerDied","Data":"21bdd4729c2dbb38cb4f0355ef86733ce9b8b83cd2cd93cb2eb79dd0c2fbdb0b"}
Nov 01 20:15:22.111197 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.111089    5066 scope.go:122] "RemoveContainer" containerID="b917f9ffdab63d77c4507529407a47c097ca8e5f0fb744622de23341ebdb9b99"
Nov 01 20:15:22.274306 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.274269    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:22.323049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.323007    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cb8d4e-29892735-wfh2t"]
Nov 01 20:15:22.731559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:15:22.731524    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" path="/var/lib/kubelet/pods/9193dfec-ed7e-4686-a413-76ddf6e0a1e7/volumes"
Nov 01 20:15:51.629879 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:15:51.629836    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:16:08.669800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.669764    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:16:08.670357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670342    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670358    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670386    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670391    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670464    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670474    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.670516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.670482    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:08.674294 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.674276    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.736897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.736861    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.737099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.736932    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.737099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.737072    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.737228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.737212    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldkl\" (UniqueName: \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.737344 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.737242    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:16:08.737344 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.737278    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.737451 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.737390    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.783548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.783512    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"kube-root-ca.crt\""
Nov 01 20:16:08.783548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.783521    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"openshift-service-ca.crt\""
Nov 01 20:16:08.783759 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.783562    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"auth-test-model-df41b8-kserve-self-signed-certs\""
Nov 01 20:16:08.783759 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.783641    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"default-dockercfg-rp5nq\""
Nov 01 20:16:08.842797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.842756    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.842869    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldkl\" (UniqueName: \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.842933    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.842977    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.843031    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.843077    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.843625 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.843575    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.845578 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.843840    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.845578 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.844346    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.851569 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.851510    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.852853 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.852829    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.901028 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.900998    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldkl\" (UniqueName: \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\") pod \"auth-test-model-df41b8-kserve-78f78cb57-r8hck\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:08.987163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:08.987129    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:09.362539 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:09.362511    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:16:09.363758 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:16:09.363720    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e652c6_d828_4cf9_9c06_c84ac36f223e.slice/crio-c59232c98049aea34dbd3d257468d078ee61b01dfdeecaab7f299c48a77fe5f6 WatchSource:0}: Error finding container c59232c98049aea34dbd3d257468d078ee61b01dfdeecaab7f299c48a77fe5f6: Status 404 returned error can't find the container with id c59232c98049aea34dbd3d257468d078ee61b01dfdeecaab7f299c48a77fe5f6
Nov 01 20:16:09.418456 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:09.418417    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" event={"ID":"d9e652c6-d828-4cf9-9c06-c84ac36f223e","Type":"ContainerStarted","Data":"c59232c98049aea34dbd3d257468d078ee61b01dfdeecaab7f299c48a77fe5f6"}
Nov 01 20:16:10.427811 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:10.427778    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" event={"ID":"d9e652c6-d828-4cf9-9c06-c84ac36f223e","Type":"ContainerStarted","Data":"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"}
Nov 01 20:16:10.541433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:10.541360    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" podStartSLOduration=2.5413444099999998 podStartE2EDuration="2.54134441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:10.540955001 +0000 UTC m=+823.925852591" watchObservedRunningTime="2026-11-01 20:16:10.54134441 +0000 UTC m=+823.926242011"
Nov 01 20:16:10.987833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:10.987801    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:12.651028 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:12.650988    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:16:18.987961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:18.987922    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:19.001549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:19.001524    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:19.516168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:19.516137    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:16:21.277803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.277765    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:16:21.278403 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.278383    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:21.278403 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.278401    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9193dfec-ed7e-4686-a413-76ddf6e0a1e7" containerName="cleanup"
Nov 01 20:16:21.281999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.281979    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.352111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.352076    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7d5118-openshift-default-dockercfg-2rtxd\""
Nov 01 20:16:21.384284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384250    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384292    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384332    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384356    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384405    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384440    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsqb\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384805 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384784    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384876    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.384985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.384915    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.398455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.398418    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:16:21.486301 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486263    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.486536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486314    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.486536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486365    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.486536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486397    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.486536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486455    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.486536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486506    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsqb\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486540    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486633    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.486675    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.487039    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.487153    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.487193    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.487808    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.489096    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.489914 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.489889    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.493800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.493755    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.544898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.544822    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.545055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.544954    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsqb\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.597753 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:21.597720    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:21.637702 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:16:21.637664    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:16:22.032409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.032385    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:16:22.034278 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:16:22.034249    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2906cec_f4c8_4191_b39f_1758f1adb4e0.slice/crio-049c69c303a77a7a3aa87b8c62cfd249316c0815bfe72b82990520b0e7573338 WatchSource:0}: Error finding container 049c69c303a77a7a3aa87b8c62cfd249316c0815bfe72b82990520b0e7573338: Status 404 returned error can't find the container with id 049c69c303a77a7a3aa87b8c62cfd249316c0815bfe72b82990520b0e7573338
Nov 01 20:16:22.036715 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.036685    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:22.036799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.036748    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:22.036799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.036779    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:22.532382 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.532347    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerStarted","Data":"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"}
Nov 01 20:16:22.532783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.532409    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerStarted","Data":"049c69c303a77a7a3aa87b8c62cfd249316c0815bfe72b82990520b0e7573338"}
Nov 01 20:16:22.598246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.598215    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:22.698015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:22.697800    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podStartSLOduration=1.697786372 podStartE2EDuration="1.697786372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:22.696776595 +0000 UTC m=+836.081674185" watchObservedRunningTime="2026-11-01 20:16:22.697786372 +0000 UTC m=+836.082683963"
Nov 01 20:16:23.604467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:23.604435    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:24.546254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:24.546225    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:24.547248 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:24.547220    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:16:27.672174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:27.672136    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerName="main" containerID="cri-o://a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3" gracePeriod=45
Nov 01 20:16:27.946032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:27.946008    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066039    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\" (UniqueName: \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066130    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066340    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066427    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066485    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.066768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.066526    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\") pod \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\" (UID: \"efeb7e65-9f4a-424c-bedd-1943c98f3aaa\") "
Nov 01 20:16:28.067586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.067555    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:28.068035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.068008    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:28.068403 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.068381    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:28.071000 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.070967    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:16:28.072744 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.072722    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "kube-api-access-xkr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:28.073071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.073043    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs" pod "efeb7e65-9f4a-424c-bedd-1943c98f3aaa" (UID: "efeb7e65-9f4a-424c-bedd-1943c98f3aaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:16:28.167961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.167922    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkr4s\" (UniqueName: \"kubernetes.io/projected/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-kube-api-access-xkr4s\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.167961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.167954    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.168391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.167969    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.168391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.168013    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.168391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.168024    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.168391 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.168036    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efeb7e65-9f4a-424c-bedd-1943c98f3aaa-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:16:28.571917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.571881    5066 generic.go:356] "Generic (PLEG): container finished" podID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerID="a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3" exitCode=0
Nov 01 20:16:28.572100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.571937    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"
Nov 01 20:16:28.572100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.571958    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" event={"ID":"efeb7e65-9f4a-424c-bedd-1943c98f3aaa","Type":"ContainerDied","Data":"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"}
Nov 01 20:16:28.572100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.572009    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" event={"ID":"efeb7e65-9f4a-424c-bedd-1943c98f3aaa","Type":"ContainerDied","Data":"94a101c7827ccb6bbcbc063254cbbe7a59a2f6486ec285c4cd8c74aadfe19b62"}
Nov 01 20:16:28.572100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.572028    5066 scope.go:122] "RemoveContainer" containerID="a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"
Nov 01 20:16:28.582223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.582205    5066 scope.go:122] "RemoveContainer" containerID="a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"
Nov 01 20:16:28.582561 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:16:28.582536    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3\": container with ID starting with a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3 not found: ID does not exist" containerID="a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"
Nov 01 20:16:28.582651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.582575    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3"} err="failed to get container status \"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3\": rpc error: code = NotFound desc = could not find container \"a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3\": container with ID starting with a34f97b1b9db8b451e3506f78704f6c04c7ed58963d9afa474cc8b01d05ac8d3 not found: ID does not exist"
Nov 01 20:16:28.734353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.734318    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:16:28.739736 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.739709    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h"]
Nov 01 20:16:28.902421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:28.902330    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-381df3/praxis-route-381df3-kserve-67cd57b67f-c5f7h" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 01 20:16:29.090528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.090474    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:16:29.091317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.091295    5066 cpu_manager.go:414] "containerMap: removing container" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerName="main"
Nov 01 20:16:29.091317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.091315    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerName="main"
Nov 01 20:16:29.091467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.091411    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" containerName="main"
Nov 01 20:16:29.096199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.096182    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.147996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.147963    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7d5118\""
Nov 01 20:16:29.150365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.150341    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:16:29.192719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.192645    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.192719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.192679    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxzj\" (UniqueName: \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.261146 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.261107    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:16:29.265585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.265563    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.294178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.294149    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.294178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.294181    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxzj\" (UniqueName: \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.294917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.294892    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.315590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.315537    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:16:29.317922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.317898    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7d5118-dockercfg-4cvn8\""
Nov 01 20:16:29.350668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.350635    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxzj\" (UniqueName: \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\") pod \"payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.395098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.395064    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.395098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.395099    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4g7x\" (UniqueName: \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.409853 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.409821    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:29.499040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.499006    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.499227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.499052    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4g7x\" (UniqueName: \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.500125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.500092    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.558652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.558613    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4g7x\" (UniqueName: \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\") pod \"payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.581223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.581196    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:29.788530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.788482    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:16:29.789235 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:16:29.789210    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975a4985_b1d8_4673_8bea_1a5ac81aff57.slice/crio-e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380 WatchSource:0}: Error finding container e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380: Status 404 returned error can't find the container with id e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380
Nov 01 20:16:29.955446 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:29.955419    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:16:29.956730 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:16:29.956703    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92bca7b_a3d4_4838_a533_37f93c5897a7.slice/crio-345e9389f95cdff394a98ed222a6e4fcded0a8f16657010a98b8b4a1bc350f0e WatchSource:0}: Error finding container 345e9389f95cdff394a98ed222a6e4fcded0a8f16657010a98b8b4a1bc350f0e: Status 404 returned error can't find the container with id 345e9389f95cdff394a98ed222a6e4fcded0a8f16657010a98b8b4a1bc350f0e
Nov 01 20:16:30.590888 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.590836    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" event={"ID":"b92bca7b-a3d4-4838-a533-37f93c5897a7","Type":"ContainerStarted","Data":"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"}
Nov 01 20:16:30.591070 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.590948    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" event={"ID":"b92bca7b-a3d4-4838-a533-37f93c5897a7","Type":"ContainerStarted","Data":"345e9389f95cdff394a98ed222a6e4fcded0a8f16657010a98b8b4a1bc350f0e"}
Nov 01 20:16:30.591153 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.591129    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:30.592793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.592767    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" event={"ID":"975a4985-b1d8-4673-8bea-1a5ac81aff57","Type":"ContainerStarted","Data":"421a4a438a83cde31573df6f195b37dd8a0d1c46c6351a153d8f390669e25b6d"}
Nov 01 20:16:30.592926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.592810    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" event={"ID":"975a4985-b1d8-4673-8bea-1a5ac81aff57","Type":"ContainerStarted","Data":"e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380"}
Nov 01 20:16:30.592926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.592831    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:30.709026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.708960    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" podStartSLOduration=1.708941924 podStartE2EDuration="1.708941924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:30.703338187 +0000 UTC m=+844.088235778" watchObservedRunningTime="2026-11-01 20:16:30.708941924 +0000 UTC m=+844.093839516"
Nov 01 20:16:30.731623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.731589    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efeb7e65-9f4a-424c-bedd-1943c98f3aaa" path="/var/lib/kubelet/pods/efeb7e65-9f4a-424c-bedd-1943c98f3aaa/volumes"
Nov 01 20:16:30.818671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:30.818602    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" podStartSLOduration=1.8185836050000002 podStartE2EDuration="1.818583605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:30.815083345 +0000 UTC m=+844.199980949" watchObservedRunningTime="2026-11-01 20:16:30.818583605 +0000 UTC m=+844.203481198"
Nov 01 20:16:41.609585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:41.608481    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:16:41.610204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:41.609807    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:16:43.100455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.100419    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:16:43.104481 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.104458    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.156652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.156621    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-df41b8-openshift-default-dockercfg-75q9g\""
Nov 01 20:16:43.178675 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.178640    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.178883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.178695    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.178883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.178810    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjm4\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.178901    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.178930    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.179117    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.179158    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.179283    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.179355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.179311    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.211210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.211176    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:16:43.280395 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280352    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280403    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280476    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280522    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280615    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280642    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280682    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280720    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.280852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280764    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjm4\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.281058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.280994    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.281058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.281043    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.281166 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.281087    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.281293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.281274    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.281758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.281736    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.283438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.283408    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.283589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.283480    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.337118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.337083    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.337293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.337270    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjm4\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.421527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.421429    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:43.849729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.849698    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:16:43.850857 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:16:43.850832    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd828bb2f_3026_47e9_a96b_53b13430f9bf.slice/crio-8fe3c25b096c920b389d57075678534e54dd613a1e6979b6de4896fc6f59c69e WatchSource:0}: Error finding container 8fe3c25b096c920b389d57075678534e54dd613a1e6979b6de4896fc6f59c69e: Status 404 returned error can't find the container with id 8fe3c25b096c920b389d57075678534e54dd613a1e6979b6de4896fc6f59c69e
Nov 01 20:16:43.853242 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.853213    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:43.853337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.853278    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:43.853337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:43.853323    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:16:44.713754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:44.713718    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" event={"ID":"d828bb2f-3026-47e9-a96b-53b13430f9bf","Type":"ContainerStarted","Data":"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"}
Nov 01 20:16:44.714256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:44.713783    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" event={"ID":"d828bb2f-3026-47e9-a96b-53b13430f9bf","Type":"ContainerStarted","Data":"8fe3c25b096c920b389d57075678534e54dd613a1e6979b6de4896fc6f59c69e"}
Nov 01 20:16:44.894145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:44.894075    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podStartSLOduration=1.894056156 podStartE2EDuration="1.894056156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:44.839351266 +0000 UTC m=+858.224248856" watchObservedRunningTime="2026-11-01 20:16:44.894056156 +0000 UTC m=+858.278953748"
Nov 01 20:16:45.421680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:45.421646    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:45.423066 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:45.423036    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:16:45.423215 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:45.423092    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:16:46.422799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:46.422760    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:16:46.423273 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:46.422820    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:16:47.422265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:47.422227    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:16:47.422483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:47.422301    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:16:48.421938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:48.421904    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:16:48.422335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:48.421971    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:16:49.422164 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:49.422128    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:16:49.422579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:49.422179    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:16:50.618276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:50.618239    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:16:50.624524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:50.624470    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:50.625098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:50.625073    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:50.625884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:50.625863    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:16:50.722351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:50.722188    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:16:51.629376 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:16:51.629338    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:16:55.619213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:55.619171    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing" containerID="cri-o://99a47b5c3bada5bd35d8939aab5166a81198e8a2c2ad21c5a0870b47734eb0bb" gracePeriod=25
Nov 01 20:16:55.723356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:16:55.723317    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing" containerID="cri-o://1b7354789ac8c35dc7ce8e46523d3d7483c80c858072677c022d4eb93c03cc86" gracePeriod=25
Nov 01 20:17:02.552011 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.551973    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:17:02.557362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.557341    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.612928 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.612895    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:17:02.656783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.656748    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.656941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.656858    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.656941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.656913    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpfr\" (UniqueName: \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.656941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.656929    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.658286 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.658270    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7d5118-serving-cert\""
Nov 01 20:17:02.758545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.758513    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpfr\" (UniqueName: \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.758545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.758546    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.758796 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.758649    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.758796 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.758706    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.758904 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:17:02.758852    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-serving-cert: secret "maas-api-e2e-inf-b-7d5118-serving-cert" not found
Nov 01 20:17:02.758955 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:17:02.758926    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls podName:f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:03.258912911 +0000 UTC m=+876.643810480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls") pod "maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" (UID: "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5") : secret "maas-api-e2e-inf-b-7d5118-serving-cert" not found
Nov 01 20:17:02.759370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.759351    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.761229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.761206    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:02.819124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:02.819014    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpfr\" (UniqueName: \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:03.264114 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:03.264078    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:03.266509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:03.266475    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:03.470847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:03.470806    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:03.845664 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:03.845637    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:17:03.846416 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:17:03.846391    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b0c56e_38b4_4d93_a66f_5f66b2e36ec5.slice/crio-fc05038ce978532a4df96ceab1948010383c7c078809aa1f4554c5be612b801d WatchSource:0}: Error finding container fc05038ce978532a4df96ceab1948010383c7c078809aa1f4554c5be612b801d: Status 404 returned error can't find the container with id fc05038ce978532a4df96ceab1948010383c7c078809aa1f4554c5be612b801d
Nov 01 20:17:04.032899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.032864    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:17:04.033068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.032870    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:17:04.841438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.841396    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" event={"ID":"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5","Type":"ContainerStarted","Data":"4b4471278ca570ffad0e50d71f93be0679d4e117bc3e8a04ec245308367aea67"}
Nov 01 20:17:04.841646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.841458    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" event={"ID":"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5","Type":"ContainerStarted","Data":"fc05038ce978532a4df96ceab1948010383c7c078809aa1f4554c5be612b801d"}
Nov 01 20:17:04.841646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.841486    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:04.977962 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:04.977910    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" podStartSLOduration=2.127672928 podStartE2EDuration="2.977894594s" totalImagesPullingTime="850.221666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:04.975825197 +0000 UTC m=+878.360722788" watchObservedRunningTime="2026-11-01 20:17:04.977894594 +0000 UTC m=+878.362792185"
Nov 01 20:17:10.853951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:10.853753    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:17:14.033132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:14.033089    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:17:14.033623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:14.033148    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:17:14.986423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:14.986385    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:17:14.990672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:14.990651    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110012 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.109974    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110053    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110195    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79xp\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110310    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110366    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110515    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110563    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110591    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.110750 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.110695    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.192156 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.192116    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:17:15.211868 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.211835    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.211897    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79xp\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.211938    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.211963    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212031    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212073    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212096    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212142    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212191    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212619    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.212834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.212813    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.213022 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.213007    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.213183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.213168    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.213322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.213302    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.215026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.215002    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.215624 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.215604    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.289572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.289240    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.295435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.295401    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79xp\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.304971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.304936    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:15.805449 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.805398    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:17:15.829806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.829767    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:15.829917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.829843    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:15.829917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.829873    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:15.942462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.940820    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" event={"ID":"0f85f370-eea4-4d59-84fa-3749a5dad918","Type":"ContainerStarted","Data":"1849d978949ab03488d2905a0a37739372a9ced1d5a1826a7fb48c72d1f2ed0b"}
Nov 01 20:17:15.944529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.943395    5066 generic.go:356] "Generic (PLEG): container finished" podID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerID="99a47b5c3bada5bd35d8939aab5166a81198e8a2c2ad21c5a0870b47734eb0bb" exitCode=0
Nov 01 20:17:15.944529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.943590    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" event={"ID":"054b71b5-54d7-4994-8e67-299a4b67f30c","Type":"ContainerDied","Data":"99a47b5c3bada5bd35d8939aab5166a81198e8a2c2ad21c5a0870b47734eb0bb"}
Nov 01 20:17:15.946851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.946728    5066 generic.go:356] "Generic (PLEG): container finished" podID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerID="1b7354789ac8c35dc7ce8e46523d3d7483c80c858072677c022d4eb93c03cc86" exitCode=0
Nov 01 20:17:15.946851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.946806    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" event={"ID":"b8cf60d6-d0e9-46da-82db-11997b8e0eea","Type":"ContainerDied","Data":"1b7354789ac8c35dc7ce8e46523d3d7483c80c858072677c022d4eb93c03cc86"}
Nov 01 20:17:16.001520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:15.996934    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:17:16.124422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.124240    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\" (UniqueName: \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\") pod \"054b71b5-54d7-4994-8e67-299a4b67f30c\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") "
Nov 01 20:17:16.124422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.124281    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\") pod \"054b71b5-54d7-4994-8e67-299a4b67f30c\" (UID: \"054b71b5-54d7-4994-8e67-299a4b67f30c\") "
Nov 01 20:17:16.125834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.124975    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume" pod "054b71b5-54d7-4994-8e67-299a4b67f30c" (UID: "054b71b5-54d7-4994-8e67-299a4b67f30c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:16.130410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.127645    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb" pod "054b71b5-54d7-4994-8e67-299a4b67f30c" (UID: "054b71b5-54d7-4994-8e67-299a4b67f30c"). InnerVolumeSpecName "kube-api-access-mslfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:16.158478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.158445    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:17:16.225665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.225624    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslfb\" (UniqueName: \"kubernetes.io/projected/054b71b5-54d7-4994-8e67-299a4b67f30c-kube-api-access-mslfb\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:17:16.225665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.225659    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054b71b5-54d7-4994-8e67-299a4b67f30c-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:17:16.326075 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.326030    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\") pod \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") "
Nov 01 20:17:16.326378 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.326287    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\" (UniqueName: \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\") pod \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\" (UID: \"b8cf60d6-d0e9-46da-82db-11997b8e0eea\") "
Nov 01 20:17:16.326934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.326908    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume" pod "b8cf60d6-d0e9-46da-82db-11997b8e0eea" (UID: "b8cf60d6-d0e9-46da-82db-11997b8e0eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:16.329673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.329642    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb" pod "b8cf60d6-d0e9-46da-82db-11997b8e0eea" (UID: "b8cf60d6-d0e9-46da-82db-11997b8e0eea"). InnerVolumeSpecName "kube-api-access-zvshb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:16.428684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.428609    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cf60d6-d0e9-46da-82db-11997b8e0eea-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:17:16.428684 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.428639    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvshb\" (UniqueName: \"kubernetes.io/projected/b8cf60d6-d0e9-46da-82db-11997b8e0eea-kube-api-access-zvshb\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:17:16.954162 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.954127    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"
Nov 01 20:17:16.954362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.954158    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf" event={"ID":"054b71b5-54d7-4994-8e67-299a4b67f30c","Type":"ContainerDied","Data":"773c4b31a05a7990e88ca496e1baafa1db0146ae58c3286bb117817edcffd46a"}
Nov 01 20:17:16.954362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.954226    5066 scope.go:122] "RemoveContainer" containerID="99a47b5c3bada5bd35d8939aab5166a81198e8a2c2ad21c5a0870b47734eb0bb"
Nov 01 20:17:16.955839 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.955815    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95" event={"ID":"b8cf60d6-d0e9-46da-82db-11997b8e0eea","Type":"ContainerDied","Data":"dad0fd08178143558479c35292210d0ef434f3f5b908c187be3da257a8d81499"}
Nov 01 20:17:16.955917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.955791    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"
Nov 01 20:17:16.958083 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:16.958058    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" event={"ID":"0f85f370-eea4-4d59-84fa-3749a5dad918","Type":"ContainerStarted","Data":"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"}
Nov 01 20:17:17.004520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.000705    5066 scope.go:122] "RemoveContainer" containerID="1b7354789ac8c35dc7ce8e46523d3d7483c80c858072677c022d4eb93c03cc86"
Nov 01 20:17:17.141997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.141954    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podStartSLOduration=3.141938004 podStartE2EDuration="3.141938004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:17.140270446 +0000 UTC m=+890.525168062" watchObservedRunningTime="2026-11-01 20:17:17.141938004 +0000 UTC m=+890.526835596"
Nov 01 20:17:17.252552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.252450    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:17:17.259932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.259900    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-381df3-65666f497c-kvbkf"]
Nov 01 20:17:17.309328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.307884    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:17.343779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.315692    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:17.343779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.315774    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:17.405518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.404624    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:17:17.427660 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:17.427621    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-381df3-86658b6687-w4b95"]
Nov 01 20:17:18.016232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.016205    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:17:18.017054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017039    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing"
Nov 01 20:17:18.017169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017157    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing"
Nov 01 20:17:18.017261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017254    5066 cpu_manager.go:414] "containerMap: removing container" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing"
Nov 01 20:17:18.017331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017324    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing"
Nov 01 20:17:18.017527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017516    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" containerName="payload-pre-processing"
Nov 01 20:17:18.017612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.017605    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" containerName="payload-processing"
Nov 01 20:17:18.022246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.022229    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.051151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.050977    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:17:18.074923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.074891    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"auth-test-model-2cc24d-kserve-self-signed-certs\""
Nov 01 20:17:18.079862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.079836    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"kube-root-ca.crt\""
Nov 01 20:17:18.111784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.111750    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"openshift-service-ca.crt\""
Nov 01 20:17:18.118567 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.112071    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"default-dockercfg-2pqzm\""
Nov 01 20:17:18.151194 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.151167    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.151698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.151681    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.151830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.151812    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.151978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.151957    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.152140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.152115    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtrz\" (UniqueName: \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.152282 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.152262    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.253843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.253567    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254196    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254384 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254348    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254451 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254408    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254526 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254456    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254563    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtrz\" (UniqueName: \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254671    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.254775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.254674    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.255123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.255061    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.256758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.256738    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.256990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.256974    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.308598 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.307883    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:18.308598 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.307948    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:18.313516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.312350    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtrz\" (UniqueName: \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\") pod \"auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.335734 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.335702    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:18.732137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.732106    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054b71b5-54d7-4994-8e67-299a4b67f30c" path="/var/lib/kubelet/pods/054b71b5-54d7-4994-8e67-299a4b67f30c/volumes"
Nov 01 20:17:18.732803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.732783    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cf60d6-d0e9-46da-82db-11997b8e0eea" path="/var/lib/kubelet/pods/b8cf60d6-d0e9-46da-82db-11997b8e0eea/volumes"
Nov 01 20:17:18.816969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:18.816945    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:17:19.027406 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.027319    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" event={"ID":"d1a63766-4395-4951-9abe-1f934e7c615f","Type":"ContainerStarted","Data":"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"}
Nov 01 20:17:19.027406 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.027370    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" event={"ID":"d1a63766-4395-4951-9abe-1f934e7c615f","Type":"ContainerStarted","Data":"041b0ba95ee21b348268ad11c1eaeeb1ea5332b2fe7d8aed2c3cb362d4e324b6"}
Nov 01 20:17:19.141964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.141911    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" podStartSLOduration=2.141896994 podStartE2EDuration="2.141896994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:19.139035726 +0000 UTC m=+892.523933318" watchObservedRunningTime="2026-11-01 20:17:19.141896994 +0000 UTC m=+892.526794585"
Nov 01 20:17:19.305964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.305717    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:19.305964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.305779    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:19.336767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:19.336737    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:20.305934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:20.305898    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:20.306118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:20.305958    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:21.305661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:21.305624    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:21.305837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:21.305683    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:21.629178 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:17:21.629094    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:17:22.305628 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:22.305586    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:17:22.305789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:22.305644    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:17:23.407227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:23.407191    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:23.407807 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:23.407682    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:23.409033 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:23.409011    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:17:24.064725 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.064689    5066 generic.go:356] "Generic (PLEG): container finished" podID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerID="5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff" exitCode=0
Nov 01 20:17:24.064930 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.064760    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerDied","Data":"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"}
Nov 01 20:17:24.065165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.065145    5066 scope.go:122] "RemoveContainer" containerID="5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"
Nov 01 20:17:24.066658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.066632    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:24.066743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.066700    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:24.066787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.066748    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:24.546788 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:24.546749    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:17:25.092570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:25.092534    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerStarted","Data":"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"}
Nov 01 20:17:25.598321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:25.598286    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:17:25.599683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:25.599653    5066 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:17:25.599845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:25.599708    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:17:26.598297 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:26.598257    5066 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:17:26.598531 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:26.598326    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:17:26.848714 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:26.848647    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:17:26.852081 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:26.852063    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:17:27.599357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:27.599320    5066 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:17:27.599809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:27.599384    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:17:28.335867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:28.335828    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:28.349113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:28.349088    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:28.599112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:28.599023    5066 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:17:28.599370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:28.599336    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:17:29.131308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:29.131276    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:17:29.598760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:29.598724    5066 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 20:17:29.598981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:29.598781    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 20:17:30.061539 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.061484    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:17:30.067156 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.067129    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147075    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147112    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147224    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147263    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pvz\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147387    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147423    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147441    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147464    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.147707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.147587    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.167532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.167275    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:17:30.248904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.248870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.248904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.248910    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.248957    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.248974    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pvz\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249035    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249074    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249091    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249120    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249183    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249329    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249377    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249441    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249603    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.249892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.249873    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.251476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.251459    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.251882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.251864    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.305071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.305042    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.305399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.305378    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pvz\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.381158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.381083    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:30.646979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.646574    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:17:30.647151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.647038    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:17:30.648385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.648363    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:17:30.822552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.821173    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:17:30.836058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.835813    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:30.836058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.835901    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:30.836058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:30.835943    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:17:31.148516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.147950    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" event={"ID":"115a3953-6c37-497b-888b-489b2e8effe9","Type":"ContainerStarted","Data":"91c349a19556a9d3ca46eab28f8fe855c8196fd25c3347493b16db5a1f3d53e0"}
Nov 01 20:17:31.148516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.148065    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" event={"ID":"115a3953-6c37-497b-888b-489b2e8effe9","Type":"ContainerStarted","Data":"81011096ebfa1f9617a4aec0276b9e216cf74a6397d5b8a3b55aee375abb0788"}
Nov 01 20:17:31.266513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.266106    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podStartSLOduration=1.266043633 podStartE2EDuration="1.266043633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:31.263456573 +0000 UTC m=+904.648354168" watchObservedRunningTime="2026-11-01 20:17:31.266043633 +0000 UTC m=+904.650941226"
Nov 01 20:17:31.383518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.382543    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:31.383712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.383667    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:31.383790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:31.383728    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:32.387803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:32.383005    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:32.387803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:32.383082    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:33.381923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:33.381879    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:33.382185 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:33.382155    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:34.386955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:34.386905    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:34.387482 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:34.386975    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:35.382339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:35.381899    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:35.382339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:35.381958    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:36.386329 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:36.386290    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:36.386791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:36.386352    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:37.381415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:37.381378    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 20:17:37.381744 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:37.381443    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 20:17:38.459970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:38.459933    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:38.460712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:38.460692    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:38.461430 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:38.461417    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:17:51.628457 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:17:51.628421    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:17:59.972370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:59.972329    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:17:59.977889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:17:59.977871    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018326    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018368    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018536    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xpj\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018584    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018647    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018696    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018779    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018841    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.018970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.018891    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.021508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.021472    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:18:00.025470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.025452    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.070897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.070856    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124000    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124075    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124111    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124160    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124203    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124325    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124381    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124429    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124455    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124487    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124543    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvp52\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124574    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124635    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124669    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xpj\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124699    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124722    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124755    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.124809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.124819    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.127517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.125234    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.127517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.125443    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.127517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.125740    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.127517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.126148    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.128922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.128897    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.129067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.129039    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.129443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.129422    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.129615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.129485    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:18:00.187961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.187927    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xpj\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.189990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.189942    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.225519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225445    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225521    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225543    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225565    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvp52\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225584    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225621    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225736    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225778    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225812    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.225935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.225869    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.226170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.226019    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.226170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.226157    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.226320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.226304    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.226603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.226579    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.227889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.227869    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.228129 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.228109    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.281802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.281774    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.281988 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.281948    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvp52\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.292548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.292515    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:00.339184 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.339149    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:00.718039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.718002    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:18:00.761653 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:18:00.761611    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677006c1_b202_4a2e_bff8_cb6e7a5e70b2.slice/crio-e9952c2aa9f1e8829ff51e16ac28e97b1677d6dda9c5cfaeca4a0788dbd8315d WatchSource:0}: Error finding container e9952c2aa9f1e8829ff51e16ac28e97b1677d6dda9c5cfaeca4a0788dbd8315d: Status 404 returned error can't find the container with id e9952c2aa9f1e8829ff51e16ac28e97b1677d6dda9c5cfaeca4a0788dbd8315d
Nov 01 20:18:00.764675 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.764625    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:00.764806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.764718    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:00.764806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.764759    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:00.792549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.792517    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:18:00.792982 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:18:00.792660    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26ffc32_4d83_4037_938e_db94b36b3970.slice/crio-a69d1e2f714b25148aba28d300c17c548c7287c36fd01fa90c5bb81a8f378821 WatchSource:0}: Error finding container a69d1e2f714b25148aba28d300c17c548c7287c36fd01fa90c5bb81a8f378821: Status 404 returned error can't find the container with id a69d1e2f714b25148aba28d300c17c548c7287c36fd01fa90c5bb81a8f378821
Nov 01 20:18:00.795485 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.795427    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:00.795648 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.795528    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:00.795786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:00.795696    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:18:01.445994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.445948    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" event={"ID":"c26ffc32-4d83-4037-938e-db94b36b3970","Type":"ContainerStarted","Data":"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"}
Nov 01 20:18:01.446468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.446058    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" event={"ID":"c26ffc32-4d83-4037-938e-db94b36b3970","Type":"ContainerStarted","Data":"a69d1e2f714b25148aba28d300c17c548c7287c36fd01fa90c5bb81a8f378821"}
Nov 01 20:18:01.447699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.447670    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" event={"ID":"677006c1-b202-4a2e-bff8-cb6e7a5e70b2","Type":"ContainerStarted","Data":"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"}
Nov 01 20:18:01.447835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.447717    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" event={"ID":"677006c1-b202-4a2e-bff8-cb6e7a5e70b2","Type":"ContainerStarted","Data":"e9952c2aa9f1e8829ff51e16ac28e97b1677d6dda9c5cfaeca4a0788dbd8315d"}
Nov 01 20:18:01.573518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.567476    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podStartSLOduration=2.567456764 podStartE2EDuration="2.567456764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.565866105 +0000 UTC m=+934.950763708" watchObservedRunningTime="2026-11-01 20:18:01.567456764 +0000 UTC m=+934.952354358"
Nov 01 20:18:01.701575 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:01.701095    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podStartSLOduration=2.70107342 podStartE2EDuration="2.70107342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.699883158 +0000 UTC m=+935.084780750" watchObservedRunningTime="2026-11-01 20:18:01.70107342 +0000 UTC m=+935.085971015"
Nov 01 20:18:02.299810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.299774    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:02.300219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.300193    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:02.300381 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.300358    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:02.340060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.340018    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:02.341650 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.341623    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:02.341772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:02.341678    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:03.295108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:03.295055    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:03.295747 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:03.295713    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:03.339483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:03.339451    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:03.339655 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:03.339528    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:04.295518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:04.294900    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:04.295518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:04.294962    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:04.339913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:04.339871    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:04.340085 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:04.339932    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:05.295517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:05.295378    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:05.295517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:05.295441    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:05.340955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:05.339751    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:05.340955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:05.339793    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:06.302995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:06.302951    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:06.303773 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:06.303736    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:06.340374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:06.340342    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:06.340547 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:06.340395    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:07.293657 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:07.293618    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 20:18:07.293828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:07.293699    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 20:18:07.340234 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:07.340198    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 01 20:18:07.340604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:07.340257    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 01 20:18:08.344271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.344198    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:08.344724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.344700    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:08.345345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.345321    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:18:08.527057 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.527027    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:08.528092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.528066    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:08.528704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:08.528687    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:18:19.333683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.333487    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:18:19.334160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.333838    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" podUID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" containerName="maas-api" containerID="cri-o://aeaec6d54eafdfc76eb09212eccba3ea0161a322860fc0fc65cdf845fb86d4b7" gracePeriod=30
Nov 01 20:18:19.632570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.632539    5066 generic.go:356] "Generic (PLEG): container finished" podID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" containerID="aeaec6d54eafdfc76eb09212eccba3ea0161a322860fc0fc65cdf845fb86d4b7" exitCode=0
Nov 01 20:18:19.632717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.632610    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" event={"ID":"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734","Type":"ContainerDied","Data":"aeaec6d54eafdfc76eb09212eccba3ea0161a322860fc0fc65cdf845fb86d4b7"}
Nov 01 20:18:19.632717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.632661    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj" event={"ID":"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734","Type":"ContainerDied","Data":"c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc"}
Nov 01 20:18:19.632717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.632677    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15bcd3212c34cf40ef0212da4a8a21adb6f9a0a6c4966c1d18a0013804c7dcc"
Nov 01 20:18:19.634269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.634254    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:18:19.816836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.816802    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\") pod \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") "
Nov 01 20:18:19.817008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.816901    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") pod \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") "
Nov 01 20:18:19.817061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.817025    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\") pod \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") "
Nov 01 20:18:19.817099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.817075    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\" (UniqueName: \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\") pod \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\" (UID: \"71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734\") "
Nov 01 20:18:19.817426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.817396    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca" pod "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" (UID: "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:19.817831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.817660    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:19.819233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.819212    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s" pod "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" (UID: "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734"). InnerVolumeSpecName "kube-api-access-kdv4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:19.819360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.819339    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs" pod "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" (UID: "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:19.819420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.819359    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls" pod "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" (UID: "71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:19.919134 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.919049    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv4s\" (UniqueName: \"kubernetes.io/projected/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-kube-api-access-kdv4s\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:19.919134 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.919078    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:19.919134 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:19.919088    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:20.638545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:20.638513    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"
Nov 01 20:18:20.804336 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:20.804299    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:18:20.852182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:20.852145    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-381df3-6dc445fdf7-hlkbj"]
Nov 01 20:18:21.717939 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:21.717900    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:18:22.731779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:22.731742    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" path="/var/lib/kubelet/pods/71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734/volumes"
Nov 01 20:18:24.227724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.227689    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-k5ttz-lw5sq"]
Nov 01 20:18:24.228365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.228348    5066 cpu_manager.go:414] "containerMap: removing container" podUID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" containerName="maas-api"
Nov 01 20:18:24.228365 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.228365    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" containerName="maas-api"
Nov 01 20:18:24.228533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.228482    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e1b9e5-6f1e-44f4-b8a2-39eb5d9c6734" containerName="maas-api"
Nov 01 20:18:24.233148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.233131    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.281870 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.281837    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.282037 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.281969    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkm4\" (UniqueName: \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.282079 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.282046    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.282138 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.282123    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.290000 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.289971    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k5ttz-lw5sq"]
Nov 01 20:18:24.383163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383125    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.383343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383221    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkm4\" (UniqueName: \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.383343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383258    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.383343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383315    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.383754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383730    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.383934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383918    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.384012 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.383988    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.439298 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.439270    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkm4\" (UniqueName: \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\") pod \"periodic-gathering-k5ttz-lw5sq\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") " pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.548623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.548545    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:18:24.926835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:24.926164    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k5ttz-lw5sq"]
Nov 01 20:18:25.701102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:25.701062    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq" event={"ID":"b34e027c-e22b-4f36-95cc-e3c332ec7c04","Type":"ContainerStarted","Data":"fabc8ee4fe26ba928e62c90ae5a69a16cb0483beccea75cbbdf1ab1261c8aca3"}
Nov 01 20:18:25.701538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:25.701126    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq" event={"ID":"b34e027c-e22b-4f36-95cc-e3c332ec7c04","Type":"ContainerStarted","Data":"d3408bf253d1802e364c5c0119c35f4cc46f5e66963ab10ee386745fa80efe2e"}
Nov 01 20:18:33.228520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:33.226083    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq" podStartSLOduration=9.226061993 podStartE2EDuration="9.226061993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:25.818390799 +0000 UTC m=+959.203288389" watchObservedRunningTime="2026-11-01 20:18:33.226061993 +0000 UTC m=+966.610959584"
Nov 01 20:18:33.233733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:33.233695    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:18:33.356518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:33.352589    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:18:35.397210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.397140    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:18:35.397727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.397355    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerName="maas-api" containerID="cri-o://e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06" gracePeriod=30
Nov 01 20:18:35.468132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.468088    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused"
Nov 01 20:18:35.705603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.705569    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:18:35.736780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.736747    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") pod \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") "
Nov 01 20:18:35.736964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.736833    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\" (UniqueName: \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\") pod \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") "
Nov 01 20:18:35.736964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.736893    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\") pod \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") "
Nov 01 20:18:35.736964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.736915    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\") pod \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\" (UID: \"1eaad4c6-ef99-4420-bda5-c428e7e61c48\") "
Nov 01 20:18:35.737546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.737513    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca" pod "1eaad4c6-ef99-4420-bda5-c428e7e61c48" (UID: "1eaad4c6-ef99-4420-bda5-c428e7e61c48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:35.737771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.737753    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaad4c6-ef99-4420-bda5-c428e7e61c48-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:35.739123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.739090    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs" pod "1eaad4c6-ef99-4420-bda5-c428e7e61c48" (UID: "1eaad4c6-ef99-4420-bda5-c428e7e61c48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:35.739217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.739159    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls" pod "1eaad4c6-ef99-4420-bda5-c428e7e61c48" (UID: "1eaad4c6-ef99-4420-bda5-c428e7e61c48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:35.739460 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.739438    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx" pod "1eaad4c6-ef99-4420-bda5-c428e7e61c48" (UID: "1eaad4c6-ef99-4420-bda5-c428e7e61c48"). InnerVolumeSpecName "kube-api-access-x57cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:35.838544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.838489    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:35.838544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.838546    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57cx\" (UniqueName: \"kubernetes.io/projected/1eaad4c6-ef99-4420-bda5-c428e7e61c48-kube-api-access-x57cx\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:35.838749 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.838561    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eaad4c6-ef99-4420-bda5-c428e7e61c48-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:35.882226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.882186    5066 generic.go:356] "Generic (PLEG): container finished" podID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerID="e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06" exitCode=0
Nov 01 20:18:35.882425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.882238    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"
Nov 01 20:18:35.882425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.882269    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" event={"ID":"1eaad4c6-ef99-4420-bda5-c428e7e61c48","Type":"ContainerDied","Data":"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"}
Nov 01 20:18:35.882425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.882330    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k" event={"ID":"1eaad4c6-ef99-4420-bda5-c428e7e61c48","Type":"ContainerDied","Data":"bc4b32be7635fb7ceb39f2137db2ea739aad5080cd184a6586dfad187a72dfed"}
Nov 01 20:18:35.882425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.882352    5066 scope.go:122] "RemoveContainer" containerID="e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"
Nov 01 20:18:35.893691 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.893667    5066 scope.go:122] "RemoveContainer" containerID="e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"
Nov 01 20:18:35.894047 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:35.894020    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06\": container with ID starting with e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06 not found: ID does not exist" containerID="e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"
Nov 01 20:18:35.894115 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:35.894061    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06"} err="failed to get container status \"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06\": rpc error: code = NotFound desc = could not find container \"e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06\": container with ID starting with e826dc19f906ed31389c6352b5ec8f8d8a5e013301de687b8220b8572ab1eb06 not found: ID does not exist"
Nov 01 20:18:36.053865 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:36.053831    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:18:36.056907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:36.056881    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb8d4e-6d68b566c7-fxc8k"]
Nov 01 20:18:36.739940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:36.739906    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" path="/var/lib/kubelet/pods/1eaad4c6-ef99-4420-bda5-c428e7e61c48/volumes"
Nov 01 20:18:38.235166 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.235126    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" podUID="a85990ac-6933-4a19-8081-a1b45896012c" containerName="payload-processing" containerID="cri-o://694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890" gracePeriod=25
Nov 01 20:18:38.357707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.357656    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" podUID="79439885-c38c-4154-8665-d2f28b94ec32" containerName="payload-pre-processing" containerID="cri-o://7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548" gracePeriod=25
Nov 01 20:18:38.523318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.523291    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:18:38.578666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.578637    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\" (UniqueName: \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\") pod \"a85990ac-6933-4a19-8081-a1b45896012c\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") "
Nov 01 20:18:38.578820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.578785    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\") pod \"a85990ac-6933-4a19-8081-a1b45896012c\" (UID: \"a85990ac-6933-4a19-8081-a1b45896012c\") "
Nov 01 20:18:38.579256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.579218    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume" pod "a85990ac-6933-4a19-8081-a1b45896012c" (UID: "a85990ac-6933-4a19-8081-a1b45896012c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:38.581143 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.581117    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2" pod "a85990ac-6933-4a19-8081-a1b45896012c" (UID: "a85990ac-6933-4a19-8081-a1b45896012c"). InnerVolumeSpecName "kube-api-access-cvqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:38.648201 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.648173    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:18:38.680796 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.680765    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\") pod \"79439885-c38c-4154-8665-d2f28b94ec32\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") "
Nov 01 20:18:38.680969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.680846    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\" (UniqueName: \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\") pod \"79439885-c38c-4154-8665-d2f28b94ec32\" (UID: \"79439885-c38c-4154-8665-d2f28b94ec32\") "
Nov 01 20:18:38.681220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.681191    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume" pod "79439885-c38c-4154-8665-d2f28b94ec32" (UID: "79439885-c38c-4154-8665-d2f28b94ec32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:38.681409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.681388    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85990ac-6933-4a19-8081-a1b45896012c-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:38.681552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.681414    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79439885-c38c-4154-8665-d2f28b94ec32-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:38.681552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.681429    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqp2\" (UniqueName: \"kubernetes.io/projected/a85990ac-6933-4a19-8081-a1b45896012c-kube-api-access-cvqp2\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:38.683161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.683131    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94" pod "79439885-c38c-4154-8665-d2f28b94ec32" (UID: "79439885-c38c-4154-8665-d2f28b94ec32"). InnerVolumeSpecName "kube-api-access-4fr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:38.787711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.787675    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fr94\" (UniqueName: \"kubernetes.io/projected/79439885-c38c-4154-8665-d2f28b94ec32-kube-api-access-4fr94\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:38.907424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.907389    5066 generic.go:356] "Generic (PLEG): container finished" podID="79439885-c38c-4154-8665-d2f28b94ec32" containerID="7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548" exitCode=0
Nov 01 20:18:38.907636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.907435    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"
Nov 01 20:18:38.907636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.907478    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" event={"ID":"79439885-c38c-4154-8665-d2f28b94ec32","Type":"ContainerDied","Data":"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"}
Nov 01 20:18:38.907636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.907553    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z" event={"ID":"79439885-c38c-4154-8665-d2f28b94ec32","Type":"ContainerDied","Data":"711b289f41db57dece8c2535f0a2ec037820163f0751486d4d2efbc91a2695e0"}
Nov 01 20:18:38.907636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.907586    5066 scope.go:122] "RemoveContainer" containerID="7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"
Nov 01 20:18:38.908941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.908919    5066 generic.go:356] "Generic (PLEG): container finished" podID="a85990ac-6933-4a19-8081-a1b45896012c" containerID="694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890" exitCode=0
Nov 01 20:18:38.909045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.908948    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" event={"ID":"a85990ac-6933-4a19-8081-a1b45896012c","Type":"ContainerDied","Data":"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"}
Nov 01 20:18:38.909045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.908973    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"
Nov 01 20:18:38.909045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.908986    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5" event={"ID":"a85990ac-6933-4a19-8081-a1b45896012c","Type":"ContainerDied","Data":"0fc434dcaa27362b945a4cf846587a2ee1e08a45118a79eac1232d4e6a644f12"}
Nov 01 20:18:38.920797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.920773    5066 scope.go:122] "RemoveContainer" containerID="7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"
Nov 01 20:18:38.921109 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:38.921091    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548\": container with ID starting with 7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548 not found: ID does not exist" containerID="7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"
Nov 01 20:18:38.921161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.921119    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548"} err="failed to get container status \"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548\": rpc error: code = NotFound desc = could not find container \"7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548\": container with ID starting with 7d3ac2b575f552c5eaa3c51970b0623c37cc203670e0e759eae22838625be548 not found: ID does not exist"
Nov 01 20:18:38.921161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.921138    5066 scope.go:122] "RemoveContainer" containerID="694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"
Nov 01 20:18:38.930929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.930912    5066 scope.go:122] "RemoveContainer" containerID="694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"
Nov 01 20:18:38.931183 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:38.931163    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890\": container with ID starting with 694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890 not found: ID does not exist" containerID="694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"
Nov 01 20:18:38.931225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:38.931192    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890"} err="failed to get container status \"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890\": rpc error: code = NotFound desc = could not find container \"694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890\": container with ID starting with 694d9a8c57db901892029cb7c29ae2e1673647dedae9fad4fa08ffb9f98d9890 not found: ID does not exist"
Nov 01 20:18:39.099358 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:39.099276    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:18:39.109548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:39.109517    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb8d4e-74784c6b5c-z2cg5"]
Nov 01 20:18:39.217671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:39.217635    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:18:39.220590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:39.220564    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb8d4e-7494fc5fb7-frr4z"]
Nov 01 20:18:40.731588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:40.731549    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79439885-c38c-4154-8665-d2f28b94ec32" path="/var/lib/kubelet/pods/79439885-c38c-4154-8665-d2f28b94ec32/volumes"
Nov 01 20:18:40.732095 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:40.732034    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85990ac-6933-4a19-8081-a1b45896012c" path="/var/lib/kubelet/pods/a85990ac-6933-4a19-8081-a1b45896012c/volumes"
Nov 01 20:18:42.420942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:42.420906    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:18:42.421446 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:42.421169    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy" containerID="cri-o://1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74" gracePeriod=30
Nov 01 20:18:47.701983 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.701919    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804017    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804188    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804223    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804251    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804287    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804349    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804419    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804474    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.804559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.804519    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\") pod \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\" (UID: \"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e\") "
Nov 01 20:18:47.808808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.808684    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:47.830789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.809142    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:47.832171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.831808    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:47.832171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.832122    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:47.850726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.842306    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "kube-api-access-qwpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:47.850726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.848229    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:47.850726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.848916    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:47.850726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.850650    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:18:47.851398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.851371    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket" pod "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" (UID: "2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:47.852711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852689    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852713    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852728    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpqv\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-kube-api-access-qwpqv\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852741    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852755    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852768    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852780    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852795    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.852880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.852808    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:18:47.973791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.973580    5066 generic.go:356] "Generic (PLEG): container finished" podID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerID="1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74" exitCode=0
Nov 01 20:18:47.973791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.973685    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" event={"ID":"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e","Type":"ContainerDied","Data":"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"}
Nov 01 20:18:47.973791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.973739    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr" event={"ID":"2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e","Type":"ContainerDied","Data":"ea6a745085e6195fce66c19ab3db164f1342adbc7b242dc50ff431621c8d9029"}
Nov 01 20:18:47.973791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.973768    5066 scope.go:122] "RemoveContainer" containerID="1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"
Nov 01 20:18:47.974109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.973926    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"
Nov 01 20:18:47.994969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.994937    5066 scope.go:122] "RemoveContainer" containerID="1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"
Nov 01 20:18:47.995336 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:47.995274    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74\": container with ID starting with 1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74 not found: ID does not exist" containerID="1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"
Nov 01 20:18:47.995445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:47.995321    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74"} err="failed to get container status \"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74\": rpc error: code = NotFound desc = could not find container \"1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74\": container with ID starting with 1436e726dc1d2a3fc7b043c4d863ffb9a85cc832b2cff89daca5a68ca9797b74 not found: ID does not exist"
Nov 01 20:18:48.156075 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:48.153892    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:18:48.161399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:48.160949    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2bhhr"]
Nov 01 20:18:48.737714 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:18:48.737668    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" path="/var/lib/kubelet/pods/2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e/volumes"
Nov 01 20:18:51.631697 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:18:51.631628    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:19:21.643831 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:19:21.643754    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:19:21.685204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:21.685167    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:19:21.685402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:21.685367    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy" containerID="cri-o://ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564" gracePeriod=30
Nov 01 20:19:26.898530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:26.898485    5066 scope.go:122] "RemoveContainer" containerID="aeaec6d54eafdfc76eb09212eccba3ea0161a322860fc0fc65cdf845fb86d4b7"
Nov 01 20:19:26.967868 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:26.967846    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:19:27.110262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110227    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110268    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110347    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110371    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110424    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110461    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110490    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110556    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110656    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:19:27.110743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110684    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\") pod \"b540aa58-cca9-46fe-81b9-8564f2673e54\" (UID: \"b540aa58-cca9-46fe-81b9-8564f2673e54\") "
Nov 01 20:19:27.110743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110697    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:19:27.110743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110706    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:27.110906 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110869    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:19:27.111018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.110996    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:19:27.111219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.111194    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.111219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.111210    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540aa58-cca9-46fe-81b9-8564f2673e54-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.111343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.111225    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.111343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.111237    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.111343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.111246    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.112819 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.112789    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:19:27.112916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.112864    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:27.113113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.113097    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "kube-api-access-g56jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:27.113173 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.113100    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo" pod "b540aa58-cca9-46fe-81b9-8564f2673e54" (UID: "b540aa58-cca9-46fe-81b9-8564f2673e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:19:27.212812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.212732    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.212812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.212759    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.212812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.212769    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.212812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.212778    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56jx\" (UniqueName: \"kubernetes.io/projected/b540aa58-cca9-46fe-81b9-8564f2673e54-kube-api-access-g56jx\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:19:27.504240 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.504204    5066 generic.go:356] "Generic (PLEG): container finished" podID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerID="ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564" exitCode=0
Nov 01 20:19:27.504413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.504259    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"
Nov 01 20:19:27.504413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.504289    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerDied","Data":"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"}
Nov 01 20:19:27.504413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.504346    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l" event={"ID":"b540aa58-cca9-46fe-81b9-8564f2673e54","Type":"ContainerDied","Data":"5a70bdf7dc101f8d36edc70c7f544b69a17df0ef3b9623b8d5eb90c6332e2640"}
Nov 01 20:19:27.504413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.504373    5066 scope.go:122] "RemoveContainer" containerID="ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"
Nov 01 20:19:27.515368 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.515347    5066 scope.go:122] "RemoveContainer" containerID="59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"
Nov 01 20:19:27.524009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.523984    5066 scope.go:122] "RemoveContainer" containerID="ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"
Nov 01 20:19:27.524269 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:19:27.524248    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564\": container with ID starting with ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564 not found: ID does not exist" containerID="ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"
Nov 01 20:19:27.524317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.524278    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564"} err="failed to get container status \"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564\": rpc error: code = NotFound desc = could not find container \"ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564\": container with ID starting with ecbabb39c7980efd53b034cf99894dcfe70ab6f27bb9c658dea39f7c4ce94564 not found: ID does not exist"
Nov 01 20:19:27.524317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.524297    5066 scope.go:122] "RemoveContainer" containerID="59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"
Nov 01 20:19:27.524565 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:19:27.524544    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4\": container with ID starting with 59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4 not found: ID does not exist" containerID="59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"
Nov 01 20:19:27.524656 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.524569    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4"} err="failed to get container status \"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4\": rpc error: code = NotFound desc = could not find container \"59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4\": container with ID starting with 59db686b9ca01f87183294ea302dd8bccac6b498e626ac7e76635fb684921df4 not found: ID does not exist"
Nov 01 20:19:27.669918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.669886    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:19:27.717809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:27.717776    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cb8d4e-openshift-default-c6c4c864f-ksv9l"]
Nov 01 20:19:28.731372 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:28.731331    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" path="/var/lib/kubelet/pods/b540aa58-cca9-46fe-81b9-8564f2673e54/volumes"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.634566    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635398    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635417    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635455    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635463    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635488    5066 cpu_manager.go:414] "containerMap: removing container" podUID="79439885-c38c-4154-8665-d2f28b94ec32" containerName="payload-pre-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635513    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79439885-c38c-4154-8665-d2f28b94ec32" containerName="payload-pre-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635548    5066 cpu_manager.go:414] "containerMap: removing container" podUID="a85990ac-6933-4a19-8081-a1b45896012c" containerName="payload-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635556    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85990ac-6933-4a19-8081-a1b45896012c" containerName="payload-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635566    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635573    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635585    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerName="maas-api"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635591    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerName="maas-api"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635754    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635769    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="79439885-c38c-4154-8665-d2f28b94ec32" containerName="payload-pre-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635780    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaad4c6-ef99-4420-bda5-c428e7e61c48" containerName="maas-api"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635797    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85990ac-6933-4a19-8081-a1b45896012c" containerName="payload-processing"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.635806    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc53d7d-8f90-4b8d-b9bd-c84bd3ae8f1e" containerName="istio-proxy"
Nov 01 20:19:32.641520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.640020    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:32.770515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.767926    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfx9b\" (UniqueName: \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:32.770515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.768186    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:32.790476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.790444    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-99d504-dockercfg-pjs87\""
Nov 01 20:19:32.790825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.790802    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-99d504\""
Nov 01 20:19:32.839190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.839131    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 20:19:32.839462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.839441    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 20:19:32.840646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.840621    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b540aa58-cca9-46fe-81b9-8564f2673e54" containerName="istio-proxy"
Nov 01 20:19:32.849530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.847772    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 20:19:32.849530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.847886    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:32.880331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.880287    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:32.922287 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.922042    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfx9b\" (UniqueName: \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:32.958604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:32.926727    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:33.082318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.082272    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rx9\" (UniqueName: \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.086514 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.082682    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.108312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.108270    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfx9b\" (UniqueName: \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\") pod \"payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:33.143760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.143716    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:33.194592 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.190533    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.194592 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.190731    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rx9\" (UniqueName: \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.194592 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.192771    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.255516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.252220    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rx9\" (UniqueName: \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\") pod \"payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.256008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.255970    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:33.957353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:33.957311    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 20:19:34.035679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:34.035577    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" event={"ID":"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511","Type":"ContainerStarted","Data":"38c94dfd8d664490fdf0afcdc161b3d4461ac126b61a1a3933c7f4ee83efa590"}
Nov 01 20:19:34.324421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:34.324390    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 20:19:34.325237 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:19:34.325211    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0f6eb6_5d80_4626_a40a_a175b4bf0a2c.slice/crio-798dc8c3e367f766cd3a379437803eaa16f3c9f7aba888b7928e8ef1c64492f0 WatchSource:0}: Error finding container 798dc8c3e367f766cd3a379437803eaa16f3c9f7aba888b7928e8ef1c64492f0: Status 404 returned error can't find the container with id 798dc8c3e367f766cd3a379437803eaa16f3c9f7aba888b7928e8ef1c64492f0
Nov 01 20:19:35.043845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.043808    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" event={"ID":"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c","Type":"ContainerStarted","Data":"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"}
Nov 01 20:19:35.044317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.043864    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" event={"ID":"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c","Type":"ContainerStarted","Data":"798dc8c3e367f766cd3a379437803eaa16f3c9f7aba888b7928e8ef1c64492f0"}
Nov 01 20:19:35.044317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.044017    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:35.045441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.045418    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" event={"ID":"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511","Type":"ContainerStarted","Data":"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"}
Nov 01 20:19:35.045565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.045471    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:35.160024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.159963    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" podStartSLOduration=3.159948344 podStartE2EDuration="3.159948344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:35.157611745 +0000 UTC m=+1028.542509336" watchObservedRunningTime="2026-11-01 20:19:35.159948344 +0000 UTC m=+1028.544845980"
Nov 01 20:19:35.272086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:35.272028    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" podStartSLOduration=3.272010601 podStartE2EDuration="3.272010601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:35.26865168 +0000 UTC m=+1028.653549272" watchObservedRunningTime="2026-11-01 20:19:35.272010601 +0000 UTC m=+1028.656908192"
Nov 01 20:19:46.056017 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:46.055919    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 20:19:46.056017 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:46.055999    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 20:19:51.630166 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:19:51.630128    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:19:55.330856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.330755    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-29ac96"]
Nov 01 20:19:55.335008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.334985    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:19:55.391555 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.391483    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-29ac96"]
Nov 01 20:19:55.400060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.400035    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\""
Nov 01 20:19:55.428630 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.428595    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkml\" (UniqueName: \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\") pod \"test-curl-7370-29ac96\" (UID: \"0bde87a9-34b4-4f59-9633-60fdcf9f31ac\") " pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:19:55.530049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.530015    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkml\" (UniqueName: \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\") pod \"test-curl-7370-29ac96\" (UID: \"0bde87a9-34b4-4f59-9633-60fdcf9f31ac\") " pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:19:55.592794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.592723    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkml\" (UniqueName: \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\") pod \"test-curl-7370-29ac96\" (UID: \"0bde87a9-34b4-4f59-9633-60fdcf9f31ac\") " pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:19:55.647280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:55.647252    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:19:56.027678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.027653    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-29ac96"]
Nov 01 20:19:56.029191 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:19:56.029158    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bde87a9_34b4_4f59_9633_60fdcf9f31ac.slice/crio-b651d9a1501f7b3f32e6fb46aaf70e40bd76f5e687956151ca81621c9dd23194 WatchSource:0}: Error finding container b651d9a1501f7b3f32e6fb46aaf70e40bd76f5e687956151ca81621c9dd23194: Status 404 returned error can't find the container with id b651d9a1501f7b3f32e6fb46aaf70e40bd76f5e687956151ca81621c9dd23194
Nov 01 20:19:56.031225 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.031204    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:19:56.202189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.202150    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-29ac96" event={"ID":"0bde87a9-34b4-4f59-9633-60fdcf9f31ac","Type":"ContainerStarted","Data":"b651d9a1501f7b3f32e6fb46aaf70e40bd76f5e687956151ca81621c9dd23194"}
Nov 01 20:19:56.273754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.273722    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:19:56.278978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.278921    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.333986 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.333922    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ad7edb-openshift-default-dockercfg-hkt6v\""
Nov 01 20:19:56.339085 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339058    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339202    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339290 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339275    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339372 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339343    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgnnc\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339403    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339573 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339551    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339595    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339707 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339672    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.339839 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.339766    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.386552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.386523    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:19:56.441100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441069    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441105    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgnnc\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441130    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441177    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441200    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441219    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441290    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441343    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441420    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441589    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441615    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441864 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441818    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.441864 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.441853    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.442188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.442170    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.443882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.443857    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.444022 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.443863    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.498371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.498342    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgnnc\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.498641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.498625    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\") pod \"e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") " pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:56.592940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:56.592853    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:57.024874 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.024845    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:19:57.025828 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:19:57.025796    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a661cb_d33b_492a_b991_99940793995b.slice/crio-9dd716bb1f024c9a82975065d1ef7109e9d1d42b45ae741e054b6dcefeab53c3 WatchSource:0}: Error finding container 9dd716bb1f024c9a82975065d1ef7109e9d1d42b45ae741e054b6dcefeab53c3: Status 404 returned error can't find the container with id 9dd716bb1f024c9a82975065d1ef7109e9d1d42b45ae741e054b6dcefeab53c3
Nov 01 20:19:57.032370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.032333    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:19:57.032464 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.032412    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:19:57.032464 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.032451    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:19:57.209921 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.209882    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" event={"ID":"d4a661cb-d33b-492a-b991-99940793995b","Type":"ContainerStarted","Data":"6b6b44d5bcbf834cde1c14de7dea4c7873ad15f5ba143052dd5e186de1fe78d4"}
Nov 01 20:19:57.210114 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.209963    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" event={"ID":"d4a661cb-d33b-492a-b991-99940793995b","Type":"ContainerStarted","Data":"9dd716bb1f024c9a82975065d1ef7109e9d1d42b45ae741e054b6dcefeab53c3"}
Nov 01 20:19:57.325975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.325863    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podStartSLOduration=1.3258487620000001 podStartE2EDuration="1.325848762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:57.32416743 +0000 UTC m=+1050.709065020" watchObservedRunningTime="2026-11-01 20:19:57.325848762 +0000 UTC m=+1050.710746352"
Nov 01 20:19:57.594704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:57.594605    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:19:58.594616 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:58.594565    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:19:58.594827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:58.594628    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:19:59.593207 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:59.593173    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:19:59.593410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:19:59.593225    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:20:00.239407 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:00.239370    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-29ac96" event={"ID":"0bde87a9-34b4-4f59-9633-60fdcf9f31ac","Type":"ContainerStarted","Data":"43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685"}
Nov 01 20:20:00.354137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:00.354075    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-29ac96" podStartSLOduration=1.949499607 podStartE2EDuration="5.354058209s" totalImagesPullingTime="3.404558602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:00.349995718 +0000 UTC m=+1053.734893309" watchObservedRunningTime="2026-11-01 20:20:00.354058209 +0000 UTC m=+1053.738955801"
Nov 01 20:20:00.593890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:00.593799    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:20:00.593890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:00.593862    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:20:01.597941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:01.597898    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:20:01.604133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:01.604081    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:20:01.869232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:01.869142    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-29ac96"]
Nov 01 20:20:01.891794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:01.891720    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-29ac96"]
Nov 01 20:20:02.641797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:02.641752    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-29ac96" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" containerName="test-curl-7370-29ac96" containerID="cri-o://43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685" gracePeriod=2
Nov 01 20:20:02.642460 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:02.642432    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:20:02.642662 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:02.642637    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:20:02.693291 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:02.693232    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-29ac96\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:03.596941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:03.596891    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 20:20:03.597280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:03.597233    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 20:20:04.700269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.700230    5066 generic.go:356] "Generic (PLEG): container finished" podID="b34e027c-e22b-4f36-95cc-e3c332ec7c04" containerID="fabc8ee4fe26ba928e62c90ae5a69a16cb0483beccea75cbbdf1ab1261c8aca3" exitCode=0
Nov 01 20:20:04.700732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.700367    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq" event={"ID":"b34e027c-e22b-4f36-95cc-e3c332ec7c04","Type":"ContainerDied","Data":"fabc8ee4fe26ba928e62c90ae5a69a16cb0483beccea75cbbdf1ab1261c8aca3"}
Nov 01 20:20:04.838639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.838611    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:20:04.839136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.839109    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:20:04.840640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.840616    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:20:04.954280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:04.954251    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:05.021080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.021048    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\" (UniqueName: \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\") pod \"0bde87a9-34b4-4f59-9633-60fdcf9f31ac\" (UID: \"0bde87a9-34b4-4f59-9633-60fdcf9f31ac\") "
Nov 01 20:20:05.023263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.023238    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml" pod "0bde87a9-34b4-4f59-9633-60fdcf9f31ac" (UID: "0bde87a9-34b4-4f59-9633-60fdcf9f31ac"). InnerVolumeSpecName "kube-api-access-fnkml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:05.122641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.122608    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkml\" (UniqueName: \"kubernetes.io/projected/0bde87a9-34b4-4f59-9633-60fdcf9f31ac-kube-api-access-fnkml\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:05.377297 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.377255    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 20:20:05.378239 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.378215    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" containerName="test-curl-7370-29ac96"
Nov 01 20:20:05.378314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.378244    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" containerName="test-curl-7370-29ac96"
Nov 01 20:20:05.378420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.378406    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" containerName="test-curl-7370-29ac96"
Nov 01 20:20:05.382586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.382555    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.426197 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.426148    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qxv\" (UniqueName: \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.426197 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.426198    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.426434 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.426403    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.426541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.426527    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.434463 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.434433    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-99d504-serving-cert\""
Nov 01 20:20:05.435790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.435763    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 20:20:05.527753 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.527713    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.527953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.527829    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.527953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.527925    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.528077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.528042    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qxv\" (UniqueName: \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.528414 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:05.528391    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-serving-cert: secret "maas-api-e2e-extmod-a-99d504-serving-cert" not found
Nov 01 20:20:05.528538 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:05.528456    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls podName:c5d4897f-7ea5-4cf2-a861-6a35f1979dfb nodeName:}" failed. No retries permitted until 2026-11-01 20:20:06.028437126 +0000 UTC m=+1059.413334709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls") pod "maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" (UID: "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb") : secret "maas-api-e2e-extmod-a-99d504-serving-cert" not found
Nov 01 20:20:05.528627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.528587    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.531154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.531125    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.585178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.585114    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qxv\" (UniqueName: \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:05.710850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.710760    5066 generic.go:356] "Generic (PLEG): container finished" podID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" containerID="43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685" exitCode=137
Nov 01 20:20:05.710850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.710804    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:05.711310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.710850    5066 scope.go:122] "RemoveContainer" containerID="43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685"
Nov 01 20:20:05.723760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.723738    5066 scope.go:122] "RemoveContainer" containerID="43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685"
Nov 01 20:20:05.724078 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:05.724055    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685\": container with ID starting with 43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685 not found: ID does not exist" containerID="43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685"
Nov 01 20:20:05.724152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.724088    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685"} err="failed to get container status \"43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685\": rpc error: code = NotFound desc = could not find container \"43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685\": container with ID starting with 43aec9f62c6605379ad7e6190de7eaeb55147f18ebde31cace707a48cad29685 not found: ID does not exist"
Nov 01 20:20:05.778300 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:05.778147    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-29ac96\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:05.851565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.851538    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:20:05.932332 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932296    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\") pod \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") "
Nov 01 20:20:05.932530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932378    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\") pod \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") "
Nov 01 20:20:05.932530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932471    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\") pod \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") "
Nov 01 20:20:05.932660 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932578    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\" (UniqueName: \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\") pod \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\" (UID: \"b34e027c-e22b-4f36-95cc-e3c332ec7c04\") "
Nov 01 20:20:05.932713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932648    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle" pod "b34e027c-e22b-4f36-95cc-e3c332ec7c04" (UID: "b34e027c-e22b-4f36-95cc-e3c332ec7c04"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:05.932713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932698    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path" pod "b34e027c-e22b-4f36-95cc-e3c332ec7c04" (UID: "b34e027c-e22b-4f36-95cc-e3c332ec7c04"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:05.932885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.932857    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle" pod "b34e027c-e22b-4f36-95cc-e3c332ec7c04" (UID: "b34e027c-e22b-4f36-95cc-e3c332ec7c04"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:05.933197 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.933179    5066 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e027c-e22b-4f36-95cc-e3c332ec7c04-archives-path\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:05.933331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.933199    5066 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-trusted-ca-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:05.933331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.933210    5066 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e027c-e22b-4f36-95cc-e3c332ec7c04-service-ca-bundle\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:05.934726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:05.934703    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4" pod "b34e027c-e22b-4f36-95cc-e3c332ec7c04" (UID: "b34e027c-e22b-4f36-95cc-e3c332ec7c04"). InnerVolumeSpecName "kube-api-access-7rkm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:06.021704 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:06.021654    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-29ac96\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:06.034518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.034465    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:06.034680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.034604    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkm4\" (UniqueName: \"kubernetes.io/projected/b34e027c-e22b-4f36-95cc-e3c332ec7c04-kube-api-access-7rkm4\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:06.036917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.036894    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:06.302262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.302160    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:06.677171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.677141    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 20:20:06.678818 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:20:06.678793    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d4897f_7ea5_4cf2_a861_6a35f1979dfb.slice/crio-49b29844e33f6b97d36dc19c8c75cd64652ee208ab0011d03f087923c1b80170 WatchSource:0}: Error finding container 49b29844e33f6b97d36dc19c8c75cd64652ee208ab0011d03f087923c1b80170: Status 404 returned error can't find the container with id 49b29844e33f6b97d36dc19c8c75cd64652ee208ab0011d03f087923c1b80170
Nov 01 20:20:06.719203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.719163    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" event={"ID":"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb","Type":"ContainerStarted","Data":"49b29844e33f6b97d36dc19c8c75cd64652ee208ab0011d03f087923c1b80170"}
Nov 01 20:20:06.721354 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.721327    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq"
Nov 01 20:20:06.721354 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.721342    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k5ttz-lw5sq" event={"ID":"b34e027c-e22b-4f36-95cc-e3c332ec7c04","Type":"ContainerDied","Data":"d3408bf253d1802e364c5c0119c35f4cc46f5e66963ab10ee386745fa80efe2e"}
Nov 01 20:20:06.721533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.721381    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3408bf253d1802e364c5c0119c35f4cc46f5e66963ab10ee386745fa80efe2e"
Nov 01 20:20:06.733409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:06.733381    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" path="/var/lib/kubelet/pods/0bde87a9-34b4-4f59-9633-60fdcf9f31ac/volumes"
Nov 01 20:20:06.780098 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:06.780060    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-29ac96\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="0bde87a9-34b4-4f59-9633-60fdcf9f31ac" pod="opendatahub/test-curl-7370-29ac96"
Nov 01 20:20:08.738424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:08.738389    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" event={"ID":"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb","Type":"ContainerStarted","Data":"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"}
Nov 01 20:20:08.738852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:08.738533    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:08.853425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:08.853364    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" podStartSLOduration=2.889961236 podStartE2EDuration="3.853350059s" totalImagesPullingTime="963.388823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:08.851227279 +0000 UTC m=+1062.236124883" watchObservedRunningTime="2026-11-01 20:20:08.853350059 +0000 UTC m=+1062.238247649"
Nov 01 20:20:09.797537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.782618    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:20:09.797537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.783511    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b34e027c-e22b-4f36-95cc-e3c332ec7c04" containerName="insights-gathering"
Nov 01 20:20:09.797537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.783533    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34e027c-e22b-4f36-95cc-e3c332ec7c04" containerName="insights-gathering"
Nov 01 20:20:09.797537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.789758    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34e027c-e22b-4f36-95cc-e3c332ec7c04" containerName="insights-gathering"
Nov 01 20:20:09.797537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.794120    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.843796 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.843768    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:20:09.846709 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.846571    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc24d-openshift-default-dockercfg-b4lxl\""
Nov 01 20:20:09.887957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.887928    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.887978    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888177 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888121    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888177 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888149    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888176    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888226    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888333    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888389    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsx7z\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.888486 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.888441    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989102    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989138    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989411 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989163    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989411 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989315    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989425    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989540    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989546    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsx7z\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989601    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989645    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989695    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989794    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.989983 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989961    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.990045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.989989    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.990088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.990051    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.991463 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.991439    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:09.992044 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:09.992027    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:10.047952 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.047877    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:10.048398 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.048378    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsx7z\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:10.115696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.115660    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:10.548440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.548405    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:20:10.549256 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:20:10.549231    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddd40f0_bd75_4c34_a188_203fde318804.slice/crio-71647fd84619f64749d91ebc1947a39d5a35ebe8ec68358a51069c3a4323ff7a WatchSource:0}: Error finding container 71647fd84619f64749d91ebc1947a39d5a35ebe8ec68358a51069c3a4323ff7a: Status 404 returned error can't find the container with id 71647fd84619f64749d91ebc1947a39d5a35ebe8ec68358a51069c3a4323ff7a
Nov 01 20:20:10.551813 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.551780    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:20:10.551903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.551853    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:20:10.551903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.551899    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:20:10.752349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.752309    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" event={"ID":"1ddd40f0-bd75-4c34-a188-203fde318804","Type":"ContainerStarted","Data":"49099708c7abb90e74db90199880722d7052d7f8ee73875c1c4461b6951b4b20"}
Nov 01 20:20:10.752563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.752370    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" event={"ID":"1ddd40f0-bd75-4c34-a188-203fde318804","Type":"ContainerStarted","Data":"71647fd84619f64749d91ebc1947a39d5a35ebe8ec68358a51069c3a4323ff7a"}
Nov 01 20:20:10.875008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:10.874902    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podStartSLOduration=1.874882722 podStartE2EDuration="1.874882722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:10.87012158 +0000 UTC m=+1064.255019174" watchObservedRunningTime="2026-11-01 20:20:10.874882722 +0000 UTC m=+1064.259780319"
Nov 01 20:20:11.115831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:11.115790    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:11.117815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:11.117787    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:11.117975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:11.117844    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:12.116546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:12.116508    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:12.117043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:12.116573    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:13.116672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:13.116631    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:13.117077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:13.116688    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:14.116440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:14.116394    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:14.116623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:14.116471    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:14.750214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:14.750183    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 20:20:15.116423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:15.116330    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:15.116423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:15.116385    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:16.116682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:16.116635    5066 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:20:16.117198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:16.116704    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:20:17.196689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:17.196648    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:17.197169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:17.197121    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:17.198346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:17.198324    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:20:21.805070 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:21.804993    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:20:32.095908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:32.095868    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:20:40.746139 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:40.746088    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podUID="2171233f-acc1-4ada-84ad-bfdfa7d4c834" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:47.116543 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.116501    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" podUID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" containerName="main" containerID="cri-o://907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd" gracePeriod=45
Nov 01 20:20:47.384704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.384649    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:20:47.518918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.518884    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.518971    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519043    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519098    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\" (UniqueName: \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519200    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519233    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\") pod \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\" (UID: \"d9e652c6-d828-4cf9-9c06-c84ac36f223e\") "
Nov 01 20:20:47.519302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519252    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:47.519633 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519604    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:47.519797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519699    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:47.519931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.519898    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:47.521478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.521449    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:47.521620 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.521547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "kube-api-access-6ldkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:47.521620 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.521580    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs" pod "d9e652c6-d828-4cf9-9c06-c84ac36f223e" (UID: "d9e652c6-d828-4cf9-9c06-c84ac36f223e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:47.620911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.620874    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:47.620911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.620909    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e652c6-d828-4cf9-9c06-c84ac36f223e-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:47.620911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.620922    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ldkl\" (UniqueName: \"kubernetes.io/projected/d9e652c6-d828-4cf9-9c06-c84ac36f223e-kube-api-access-6ldkl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:47.621173 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.620936    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:47.621173 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:47.620949    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e652c6-d828-4cf9-9c06-c84ac36f223e-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:48.174397 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.174350    5066 generic.go:356] "Generic (PLEG): container finished" podID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" containerID="907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd" exitCode=0
Nov 01 20:20:48.174397 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.174400    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"
Nov 01 20:20:48.174904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.174445    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" event={"ID":"d9e652c6-d828-4cf9-9c06-c84ac36f223e","Type":"ContainerDied","Data":"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"}
Nov 01 20:20:48.174904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.174531    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck" event={"ID":"d9e652c6-d828-4cf9-9c06-c84ac36f223e","Type":"ContainerDied","Data":"c59232c98049aea34dbd3d257468d078ee61b01dfdeecaab7f299c48a77fe5f6"}
Nov 01 20:20:48.174904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.174557    5066 scope.go:122] "RemoveContainer" containerID="907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"
Nov 01 20:20:48.185694 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.185673    5066 scope.go:122] "RemoveContainer" containerID="907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"
Nov 01 20:20:48.185972 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:48.185953    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd\": container with ID starting with 907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd not found: ID does not exist" containerID="907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"
Nov 01 20:20:48.186030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.185983    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd"} err="failed to get container status \"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd\": rpc error: code = NotFound desc = could not find container \"907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd\": container with ID starting with 907c65fde55c30ec656099c4dad87a87f56d2b3b482b2a0009725aabd26875bd not found: ID does not exist"
Nov 01 20:20:48.337899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.337861    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:20:48.386355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.386324    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/auth-test-model-df41b8-kserve-78f78cb57-r8hck"]
Nov 01 20:20:48.731812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:48.731776    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" path="/var/lib/kubelet/pods/d9e652c6-d828-4cf9-9c06-c84ac36f223e/volumes"
Nov 01 20:20:50.718249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.718209    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:20:50.814076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.814037    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-91296d"]
Nov 01 20:20:50.815634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.815604    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" containerName="main"
Nov 01 20:20:50.815797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.815784    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" containerName="main"
Nov 01 20:20:50.816261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.816241    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e652c6-d828-4cf9-9c06-c84ac36f223e" containerName="main"
Nov 01 20:20:50.820995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.820964    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:50.859969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.859929    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-91296d"]
Nov 01 20:20:50.873026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.872985    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\""
Nov 01 20:20:50.962514 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:50.962458    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4ql\" (UniqueName: \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\") pod \"test-curl-7370-91296d\" (UID: \"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f\") " pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:51.064250 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:51.064161    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4ql\" (UniqueName: \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\") pod \"test-curl-7370-91296d\" (UID: \"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f\") " pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:51.120582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:51.120548    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4ql\" (UniqueName: \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\") pod \"test-curl-7370-91296d\" (UID: \"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f\") " pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:51.135405 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:51.135374    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:51.509930 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:51.509897    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-91296d"]
Nov 01 20:20:51.510367 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:20:51.510333    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf752aa1f_57ed_467f_9ef5_5f6a66e1f63f.slice/crio-d01753d6f212be0ed2473916507b32792d3c60967394a01009b7f83353389cbd WatchSource:0}: Error finding container d01753d6f212be0ed2473916507b32792d3c60967394a01009b7f83353389cbd: Status 404 returned error can't find the container with id d01753d6f212be0ed2473916507b32792d3c60967394a01009b7f83353389cbd
Nov 01 20:20:51.630037 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:51.630003    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:20:52.207099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:52.207064    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-91296d" event={"ID":"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f","Type":"ContainerStarted","Data":"cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb"}
Nov 01 20:20:52.207587 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:52.207112    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-91296d" event={"ID":"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f","Type":"ContainerStarted","Data":"d01753d6f212be0ed2473916507b32792d3c60967394a01009b7f83353389cbd"}
Nov 01 20:20:52.319385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:52.319323    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-91296d" podStartSLOduration=2.319308499 podStartE2EDuration="2.319308499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:52.316736681 +0000 UTC m=+1105.701634275" watchObservedRunningTime="2026-11-01 20:20:52.319308499 +0000 UTC m=+1105.704206090"
Nov 01 20:20:53.673077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.673041    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-91296d"]
Nov 01 20:20:53.682815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.682784    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-91296d"]
Nov 01 20:20:53.819255 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.819220    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-91284f"]
Nov 01 20:20:53.819942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.819924    5066 cpu_manager.go:414] "containerMap: removing container" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" containerName="test-curl-7370-91296d"
Nov 01 20:20:53.819997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.819944    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" containerName="test-curl-7370-91296d"
Nov 01 20:20:53.820073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.820063    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" containerName="test-curl-7370-91296d"
Nov 01 20:20:53.822431 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.822412    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:53.875546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:53.875517    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-91284f"]
Nov 01 20:20:54.002310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.002271    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknvl\" (UniqueName: \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\") pod \"test-curl-7370-91284f\" (UID: \"aef5ddf5-ee7e-437c-947a-3ce45348ec68\") " pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:54.104093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.104054    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknvl\" (UniqueName: \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\") pod \"test-curl-7370-91284f\" (UID: \"aef5ddf5-ee7e-437c-947a-3ce45348ec68\") " pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:54.160198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.160165    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknvl\" (UniqueName: \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\") pod \"test-curl-7370-91284f\" (UID: \"aef5ddf5-ee7e-437c-947a-3ce45348ec68\") " pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:54.219694 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.219638    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-91296d" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" containerName="test-curl-7370-91296d" containerID="cri-o://cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb" gracePeriod=2
Nov 01 20:20:54.270323 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:54.270241    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91296d\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:54.433566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.433534    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:54.810238 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:54.810204    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-91284f"]
Nov 01 20:20:54.811447 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:20:54.811415    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaef5ddf5_ee7e_437c_947a_3ce45348ec68.slice/crio-2f9a833eefd72b3e410d6f4f41dfdd4e72627239095b768feeef62487b7021df WatchSource:0}: Error finding container 2f9a833eefd72b3e410d6f4f41dfdd4e72627239095b768feeef62487b7021df: Status 404 returned error can't find the container with id 2f9a833eefd72b3e410d6f4f41dfdd4e72627239095b768feeef62487b7021df
Nov 01 20:20:55.227886 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:55.227846    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-91284f" event={"ID":"aef5ddf5-ee7e-437c-947a-3ce45348ec68","Type":"ContainerStarted","Data":"bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46"}
Nov 01 20:20:55.227886 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:55.227889    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-91284f" event={"ID":"aef5ddf5-ee7e-437c-947a-3ce45348ec68","Type":"ContainerStarted","Data":"2f9a833eefd72b3e410d6f4f41dfdd4e72627239095b768feeef62487b7021df"}
Nov 01 20:20:55.340175 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:55.340124    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-91284f" podStartSLOduration=2.340110943 podStartE2EDuration="2.340110943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:55.338205635 +0000 UTC m=+1108.723103226" watchObservedRunningTime="2026-11-01 20:20:55.340110943 +0000 UTC m=+1108.725008534"
Nov 01 20:20:56.465704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.465668    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:56.633397 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.633317    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\" (UniqueName: \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\") pod \"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f\" (UID: \"f752aa1f-57ed-467f-9ef5-5f6a66e1f63f\") "
Nov 01 20:20:56.635430 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.635400    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql" pod "f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" (UID: "f752aa1f-57ed-467f-9ef5-5f6a66e1f63f"). InnerVolumeSpecName "kube-api-access-fk4ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:56.733450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.733417    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" path="/var/lib/kubelet/pods/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f/volumes"
Nov 01 20:20:56.734522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.734465    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4ql\" (UniqueName: \"kubernetes.io/projected/f752aa1f-57ed-467f-9ef5-5f6a66e1f63f-kube-api-access-fk4ql\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:20:56.934566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.933790    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-91284f"]
Nov 01 20:20:56.940071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:56.940038    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-91284f"]
Nov 01 20:20:57.016002 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.015954    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-d47ec7"]
Nov 01 20:20:57.016670 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.016639    5066 cpu_manager.go:414] "containerMap: removing container" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" containerName="test-curl-7370-91284f"
Nov 01 20:20:57.016670 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.016666    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" containerName="test-curl-7370-91284f"
Nov 01 20:20:57.016908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.016891    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" containerName="test-curl-7370-91284f"
Nov 01 20:20:57.019665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.019646    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:20:57.072932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.072359    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-d47ec7"]
Nov 01 20:20:57.140149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.140116    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppzh\" (UniqueName: \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\") pod \"test-curl-7370-d47ec7\" (UID: \"fbeb6fcf-a711-4f50-94b5-5b9684e03c67\") " pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:20:57.240999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.240949    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppzh\" (UniqueName: \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\") pod \"test-curl-7370-d47ec7\" (UID: \"fbeb6fcf-a711-4f50-94b5-5b9684e03c67\") " pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:20:57.245389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.245363    5066 generic.go:356] "Generic (PLEG): container finished" podID="f752aa1f-57ed-467f-9ef5-5f6a66e1f63f" containerID="cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb" exitCode=137
Nov 01 20:20:57.245549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.245409    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91296d"
Nov 01 20:20:57.245549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.245439    5066 scope.go:122] "RemoveContainer" containerID="cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb"
Nov 01 20:20:57.245653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.245627    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-91284f" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" containerName="test-curl-7370-91284f" containerID="cri-o://bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46" gracePeriod=2
Nov 01 20:20:57.254955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.254938    5066 scope.go:122] "RemoveContainer" containerID="cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb"
Nov 01 20:20:57.255204 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:57.255185    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb\": container with ID starting with cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb not found: ID does not exist" containerID="cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb"
Nov 01 20:20:57.255262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.255216    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb"} err="failed to get container status \"cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb\": rpc error: code = NotFound desc = could not find container \"cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb\": container with ID starting with cde3bab867d0733ccb675cbf77d7896a3c8cf97a87543d5c59723bba34243dbb not found: ID does not exist"
Nov 01 20:20:57.296019 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:57.295981    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91284f\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:57.297284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.297266    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppzh\" (UniqueName: \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\") pod \"test-curl-7370-d47ec7\" (UID: \"fbeb6fcf-a711-4f50-94b5-5b9684e03c67\") " pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:20:57.331802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.331776    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:20:57.346449 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:57.346419    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91284f\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:20:57.705613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:57.705584    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-d47ec7"]
Nov 01 20:20:57.706447 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:20:57.706417    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbeb6fcf_a711_4f50_94b5_5b9684e03c67.slice/crio-4a87738c01cb163c18c2d1c216bc591e0364188e266c4af88da1fdca5f9e0121 WatchSource:0}: Error finding container 4a87738c01cb163c18c2d1c216bc591e0364188e266c4af88da1fdca5f9e0121: Status 404 returned error can't find the container with id 4a87738c01cb163c18c2d1c216bc591e0364188e266c4af88da1fdca5f9e0121
Nov 01 20:20:58.257662 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:58.257624    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-d47ec7" event={"ID":"fbeb6fcf-a711-4f50-94b5-5b9684e03c67","Type":"ContainerStarted","Data":"c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d"}
Nov 01 20:20:58.257862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:20:58.257672    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-d47ec7" event={"ID":"fbeb6fcf-a711-4f50-94b5-5b9684e03c67","Type":"ContainerStarted","Data":"4a87738c01cb163c18c2d1c216bc591e0364188e266c4af88da1fdca5f9e0121"}
Nov 01 20:20:58.450519 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:20:58.447744    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91284f\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:21:00.047745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.047675    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:21:00.246321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.246290    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\" (UniqueName: \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\") pod \"aef5ddf5-ee7e-437c-947a-3ce45348ec68\" (UID: \"aef5ddf5-ee7e-437c-947a-3ce45348ec68\") "
Nov 01 20:21:00.248504 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.248466    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl" pod "aef5ddf5-ee7e-437c-947a-3ce45348ec68" (UID: "aef5ddf5-ee7e-437c-947a-3ce45348ec68"). InnerVolumeSpecName "kube-api-access-tknvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:00.281074 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.280983    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-d47ec7" podStartSLOduration=4.280965053 podStartE2EDuration="4.280965053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:58.38977582 +0000 UTC m=+1111.774673424" watchObservedRunningTime="2026-11-01 20:21:00.280965053 +0000 UTC m=+1113.665862645"
Nov 01 20:21:00.283299 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.283271    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-d47ec7"]
Nov 01 20:21:00.286444 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.286420    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-d47ec7"]
Nov 01 20:21:00.309584 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.309480    5066 generic.go:356] "Generic (PLEG): container finished" podID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" containerID="bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46" exitCode=137
Nov 01 20:21:00.309584 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.309550    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:21:00.309793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.309595    5066 scope.go:122] "RemoveContainer" containerID="bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46"
Nov 01 20:21:00.309906 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.309880    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-d47ec7" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" containerName="test-curl-7370-d47ec7" containerID="cri-o://c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d" gracePeriod=2
Nov 01 20:21:00.324400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.324376    5066 scope.go:122] "RemoveContainer" containerID="bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46"
Nov 01 20:21:00.326319 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.326287    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46\": container with ID starting with bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46 not found: ID does not exist" containerID="bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46"
Nov 01 20:21:00.326415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.326332    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46"} err="failed to get container status \"bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46\": rpc error: code = NotFound desc = could not find container \"bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46\": container with ID starting with bb6cb4910d283a154a1e654396cc917eceb94640e365d67d9c88999e2d93fa46 not found: ID does not exist"
Nov 01 20:21:00.347748 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.347725    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknvl\" (UniqueName: \"kubernetes.io/projected/aef5ddf5-ee7e-437c-947a-3ce45348ec68-kube-api-access-tknvl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:00.360846 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.360818    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-d47ec7\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:21:00.411363 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.411326    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91284f\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:21:00.431123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.431094    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7370-22d682"]
Nov 01 20:21:00.431727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.431713    5066 cpu_manager.go:414] "containerMap: removing container" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" containerName="test-curl-7370-d47ec7"
Nov 01 20:21:00.431784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.431729    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" containerName="test-curl-7370-d47ec7"
Nov 01 20:21:00.431834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.431825    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" containerName="test-curl-7370-d47ec7"
Nov 01 20:21:00.434686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.434670    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:00.449349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.449323    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjmp\" (UniqueName: \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\") pod \"test-curl-7370-22d682\" (UID: \"f1d05324-6b50-4a76-a953-46984d80da3e\") " pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:00.461897 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.461856    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-d47ec7\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:21:00.489101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.488314    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-22d682"]
Nov 01 20:21:00.512606 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.512573    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-d47ec7\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:21:00.550300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.550270    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjmp\" (UniqueName: \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\") pod \"test-curl-7370-22d682\" (UID: \"f1d05324-6b50-4a76-a953-46984d80da3e\") " pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:00.606276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.606193    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjmp\" (UniqueName: \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\") pod \"test-curl-7370-22d682\" (UID: \"f1d05324-6b50-4a76-a953-46984d80da3e\") " pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:00.672064 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:00.672017    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-91284f\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" pod="opendatahub/test-curl-7370-91284f"
Nov 01 20:21:00.731588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.731550    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef5ddf5-ee7e-437c-947a-3ce45348ec68" path="/var/lib/kubelet/pods/aef5ddf5-ee7e-437c-947a-3ce45348ec68/volumes"
Nov 01 20:21:00.745749 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:00.745717    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:01.125112 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:21:01.125077    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d05324_6b50_4a76_a953_46984d80da3e.slice/crio-a3890b2f0a886ab5c6d50dcc8d040839d0c2675ad56e6957b98ce263d5ccdf9d WatchSource:0}: Error finding container a3890b2f0a886ab5c6d50dcc8d040839d0c2675ad56e6957b98ce263d5ccdf9d: Status 404 returned error can't find the container with id a3890b2f0a886ab5c6d50dcc8d040839d0c2675ad56e6957b98ce263d5ccdf9d
Nov 01 20:21:01.125433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:01.125210    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7370-22d682"]
Nov 01 20:21:01.317777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:01.317738    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-22d682" event={"ID":"f1d05324-6b50-4a76-a953-46984d80da3e","Type":"ContainerStarted","Data":"0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1"}
Nov 01 20:21:01.317777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:01.317782    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7370-22d682" event={"ID":"f1d05324-6b50-4a76-a953-46984d80da3e","Type":"ContainerStarted","Data":"a3890b2f0a886ab5c6d50dcc8d040839d0c2675ad56e6957b98ce263d5ccdf9d"}
Nov 01 20:21:01.430944 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:01.430832    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7370-22d682" podStartSLOduration=1.430818782 podStartE2EDuration="1.430818782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:01.428745404 +0000 UTC m=+1114.813642995" watchObservedRunningTime="2026-11-01 20:21:01.430818782 +0000 UTC m=+1114.815716373"
Nov 01 20:21:02.538470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.538439    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:21:02.572991 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.572921    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\" (UniqueName: \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\") pod \"fbeb6fcf-a711-4f50-94b5-5b9684e03c67\" (UID: \"fbeb6fcf-a711-4f50-94b5-5b9684e03c67\") "
Nov 01 20:21:02.574939 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.574917    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh" pod "fbeb6fcf-a711-4f50-94b5-5b9684e03c67" (UID: "fbeb6fcf-a711-4f50-94b5-5b9684e03c67"). InnerVolumeSpecName "kube-api-access-6ppzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:02.674511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.674466    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppzh\" (UniqueName: \"kubernetes.io/projected/fbeb6fcf-a711-4f50-94b5-5b9684e03c67-kube-api-access-6ppzh\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:02.731733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.731700    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" path="/var/lib/kubelet/pods/fbeb6fcf-a711-4f50-94b5-5b9684e03c67/volumes"
Nov 01 20:21:02.863900 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.863794    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7370-22d682"]
Nov 01 20:21:02.867231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:02.867199    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7370-22d682"]
Nov 01 20:21:03.337077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.337041    5066 generic.go:356] "Generic (PLEG): container finished" podID="fbeb6fcf-a711-4f50-94b5-5b9684e03c67" containerID="c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d" exitCode=137
Nov 01 20:21:03.337352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.337084    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-d47ec7"
Nov 01 20:21:03.337352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.337139    5066 scope.go:122] "RemoveContainer" containerID="c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d"
Nov 01 20:21:03.337352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.337254    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7370-22d682" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" containerName="test-curl-7370-22d682" containerID="cri-o://0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1" gracePeriod=2
Nov 01 20:21:03.346898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.346868    5066 scope.go:122] "RemoveContainer" containerID="c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d"
Nov 01 20:21:03.347171 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:03.347146    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d\": container with ID starting with c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d not found: ID does not exist" containerID="c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d"
Nov 01 20:21:03.347236 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:03.347180    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d"} err="failed to get container status \"c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d\": rpc error: code = NotFound desc = could not find container \"c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d\": container with ID starting with c936f9d7cd69d25829a94b9bed5bd62d09864a83a2560a6c7801f1dade310d7d not found: ID does not exist"
Nov 01 20:21:03.387886 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:03.387848    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-22d682\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:03.438364 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:03.438320    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-22d682\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:05.349528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.335717    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 20:21:05.379424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.377187    5066 cpu_manager.go:414] "containerMap: removing container" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" containerName="test-curl-7370-22d682"
Nov 01 20:21:05.379424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.377217    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" containerName="test-curl-7370-22d682"
Nov 01 20:21:05.379424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.377421    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" containerName="test-curl-7370-22d682"
Nov 01 20:21:05.398404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.398375    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.411035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.410999    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 20:21:05.451399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.451369    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6a675a-dockercfg-9hq59\""
Nov 01 20:21:05.451561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.451411    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6a675a\""
Nov 01 20:21:05.453692 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.453400    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdnz\" (UniqueName: \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.453692 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.453448    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.554287 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.554248    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdnz\" (UniqueName: \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.554287 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.554291    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.555272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.555250    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.611157 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.611091    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdnz\" (UniqueName: \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\") pod \"payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.616068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.616045    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:05.655786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.655757    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\" (UniqueName: \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\") pod \"f1d05324-6b50-4a76-a953-46984d80da3e\" (UID: \"f1d05324-6b50-4a76-a953-46984d80da3e\") "
Nov 01 20:21:05.657837 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.657812    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp" pod "f1d05324-6b50-4a76-a953-46984d80da3e" (UID: "f1d05324-6b50-4a76-a953-46984d80da3e"). InnerVolumeSpecName "kube-api-access-dtjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:05.745212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.745178    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:05.752425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.752388    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" podUID="d1a63766-4395-4951-9abe-1f934e7c615f" containerName="main" containerID="cri-o://60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275" gracePeriod=45
Nov 01 20:21:05.757786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.757669    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjmp\" (UniqueName: \"kubernetes.io/projected/f1d05324-6b50-4a76-a953-46984d80da3e-kube-api-access-dtjmp\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:05.998719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:05.998692    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:21:06.061310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061273    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\" (UniqueName: \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061488 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061402    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061488 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061450    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061534    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061609    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061655    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\") pod \"d1a63766-4395-4951-9abe-1f934e7c615f\" (UID: \"d1a63766-4395-4951-9abe-1f934e7c615f\") "
Nov 01 20:21:06.061798 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061775    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:06.061912 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.061886    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:06.062051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.062025    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:06.062231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.062214    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.062292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.062236    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.062292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.062251    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.063742 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.063721    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:21:06.064050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.064028    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "kube-api-access-zqtrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:06.064134 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.064026    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm" pod "d1a63766-4395-4951-9abe-1f934e7c615f" (UID: "d1a63766-4395-4951-9abe-1f934e7c615f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:06.126326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.126265    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 20:21:06.126453 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:21:06.126427    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292b3de2_9c16_4c09_892c_2c72992231b4.slice/crio-d041298d4cf54f36d9a6e8d8eb5642ad7c7abffeaa4dde9a3284399ccfba0f59 WatchSource:0}: Error finding container d041298d4cf54f36d9a6e8d8eb5642ad7c7abffeaa4dde9a3284399ccfba0f59: Status 404 returned error can't find the container with id d041298d4cf54f36d9a6e8d8eb5642ad7c7abffeaa4dde9a3284399ccfba0f59
Nov 01 20:21:06.163859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.163828    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a63766-4395-4951-9abe-1f934e7c615f-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.163976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.163862    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a63766-4395-4951-9abe-1f934e7c615f-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.163976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.163879    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtrz\" (UniqueName: \"kubernetes.io/projected/d1a63766-4395-4951-9abe-1f934e7c615f-kube-api-access-zqtrz\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:06.408024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.407918    5066 generic.go:356] "Generic (PLEG): container finished" podID="d1a63766-4395-4951-9abe-1f934e7c615f" containerID="60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275" exitCode=0
Nov 01 20:21:06.408024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.407974    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"
Nov 01 20:21:06.408612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.408010    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" event={"ID":"d1a63766-4395-4951-9abe-1f934e7c615f","Type":"ContainerDied","Data":"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"}
Nov 01 20:21:06.408612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.408062    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q" event={"ID":"d1a63766-4395-4951-9abe-1f934e7c615f","Type":"ContainerDied","Data":"041b0ba95ee21b348268ad11c1eaeeb1ea5332b2fe7d8aed2c3cb362d4e324b6"}
Nov 01 20:21:06.408612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.408085    5066 scope.go:122] "RemoveContainer" containerID="60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"
Nov 01 20:21:06.409433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.409413    5066 generic.go:356] "Generic (PLEG): container finished" podID="f1d05324-6b50-4a76-a953-46984d80da3e" containerID="0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1" exitCode=137
Nov 01 20:21:06.409620 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.409595    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:06.411593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.411558    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" event={"ID":"292b3de2-9c16-4c09-892c-2c72992231b4","Type":"ContainerStarted","Data":"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"}
Nov 01 20:21:06.411704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.411617    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" event={"ID":"292b3de2-9c16-4c09-892c-2c72992231b4","Type":"ContainerStarted","Data":"d041298d4cf54f36d9a6e8d8eb5642ad7c7abffeaa4dde9a3284399ccfba0f59"}
Nov 01 20:21:06.411704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.411662    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:06.420058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.420041    5066 scope.go:122] "RemoveContainer" containerID="60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"
Nov 01 20:21:06.420355 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:06.420333    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275\": container with ID starting with 60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275 not found: ID does not exist" containerID="60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"
Nov 01 20:21:06.420404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.420365    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275"} err="failed to get container status \"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275\": rpc error: code = NotFound desc = could not find container \"60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275\": container with ID starting with 60e67387db0a135fa8f145f708b4c64f1d848d0489b28883b27f86620ecc4275 not found: ID does not exist"
Nov 01 20:21:06.420404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.420386    5066 scope.go:122] "RemoveContainer" containerID="0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1"
Nov 01 20:21:06.441336 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.441319    5066 scope.go:122] "RemoveContainer" containerID="0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1"
Nov 01 20:21:06.441657 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:06.441630    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1\": container with ID starting with 0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1 not found: ID does not exist" containerID="0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1"
Nov 01 20:21:06.441716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.441669    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1"} err="failed to get container status \"0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1\": rpc error: code = NotFound desc = could not find container \"0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1\": container with ID starting with 0962c7cd00f646d39f19729ba593f8ed1ce1eab793200c9c25dc24551a6be7b1 not found: ID does not exist"
Nov 01 20:21:06.510120 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:06.510089    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-22d682\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:06.622246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.622183    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" podStartSLOduration=1.622166381 podStartE2EDuration="1.622166381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:06.621168855 +0000 UTC m=+1120.006066445" watchObservedRunningTime="2026-11-01 20:21:06.622166381 +0000 UTC m=+1120.007063974"
Nov 01 20:21:06.743856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.743823    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" path="/var/lib/kubelet/pods/f1d05324-6b50-4a76-a953-46984d80da3e/volumes"
Nov 01 20:21:06.744162 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.744146    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:21:06.744253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:06.744176    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/auth-test-model-2cc24d-kserve-5b85557b9c-5ck8q"]
Nov 01 20:21:06.793732 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:06.793694    5066 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7370-22d682\" is forbidden: User \"system:node:ip-10-0-142-223.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-223.us-west-1.compute.internal' and this object" podUID="f1d05324-6b50-4a76-a953-46984d80da3e" pod="opendatahub/test-curl-7370-22d682"
Nov 01 20:21:08.731148 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:08.731058    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a63766-4395-4951-9abe-1f934e7c615f" path="/var/lib/kubelet/pods/d1a63766-4395-4951-9abe-1f934e7c615f/volumes"
Nov 01 20:21:09.808275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.808233    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:21:09.809175 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.809154    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d1a63766-4395-4951-9abe-1f934e7c615f" containerName="main"
Nov 01 20:21:09.809226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.809179    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a63766-4395-4951-9abe-1f934e7c615f" containerName="main"
Nov 01 20:21:09.809334 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.809323    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a63766-4395-4951-9abe-1f934e7c615f" containerName="main"
Nov 01 20:21:09.814351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.814327    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.868651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.868616    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:21:09.906839 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.906804    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqtl\" (UniqueName: \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.907024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.906935    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.907079 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.907058    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.907151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.907131    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.907274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.907223    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.907333 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.907270    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:09.912367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.912335    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"distinct-w0-d204f4-kserve-self-signed-certs\""
Nov 01 20:21:09.912367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.912347    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"default-dockercfg-qdjvd\""
Nov 01 20:21:09.915272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.915257    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"openshift-service-ca.crt\""
Nov 01 20:21:09.915340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:09.915271    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"kube-root-ca.crt\""
Nov 01 20:21:10.008412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008379    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.008658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008446    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.008658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008484    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqtl\" (UniqueName: \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.008658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008561    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.008839 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008678    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.008961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.008936    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.009073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.009043    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.009198 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.009181    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.009508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.009477    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.010985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.010963    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.011376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.011351    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.066925 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.066846    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqtl\" (UniqueName: \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\") pod \"distinct-w0-d204f4-kserve-567f66fcb5-lrlpn\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") " pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.127193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.127149    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:10.526133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:10.526082    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:21:10.527609 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:21:10.527576    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb1bd6d1_46a1_4bda_a2bf_d1eb01430f52.slice/crio-d2add6daf5eab8a15c2f8873f295396a5449fa9a4478fae4e7fc074b321fc000 WatchSource:0}: Error finding container d2add6daf5eab8a15c2f8873f295396a5449fa9a4478fae4e7fc074b321fc000: Status 404 returned error can't find the container with id d2add6daf5eab8a15c2f8873f295396a5449fa9a4478fae4e7fc074b321fc000
Nov 01 20:21:11.449738 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:11.449699    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" event={"ID":"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52","Type":"ContainerStarted","Data":"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"}
Nov 01 20:21:11.449738 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:11.449743    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" event={"ID":"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52","Type":"ContainerStarted","Data":"d2add6daf5eab8a15c2f8873f295396a5449fa9a4478fae4e7fc074b321fc000"}
Nov 01 20:21:11.585807 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:11.585742    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" podStartSLOduration=2.585725134 podStartE2EDuration="2.585725134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:11.567186027 +0000 UTC m=+1124.952083619" watchObservedRunningTime="2026-11-01 20:21:11.585725134 +0000 UTC m=+1124.970622725"
Nov 01 20:21:12.127805 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:12.127771    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:17.423720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:17.423687    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 20:21:20.128246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:20.128191    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:20.158091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:20.158052    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:20.917516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:20.914836    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:21:21.884519 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:21.882293    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:21:31.601910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.601876    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:21:31.608433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.608408    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.633160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.632985    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:21:31.660951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.660915    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"default-dockercfg-rp5nq\""
Nov 01 20:21:31.661154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.661071    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"rate-test-model-df41b8-kserve-self-signed-certs\""
Nov 01 20:21:31.661229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.661145    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"kube-root-ca.crt\""
Nov 01 20:21:31.661409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.661379    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"openshift-service-ca.crt\""
Nov 01 20:21:31.741128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741084    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.741374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741137    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.741374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741344    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.741516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741385    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qfw\" (UniqueName: \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.741516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741432    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.741634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.741487    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.843933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.843893    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844126 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.843944    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.844136    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.844174    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qfw\" (UniqueName: \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.844221    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.844263    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.844876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.844847    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.845414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.845393    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.845993 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.845960    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.856635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.856539    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.862180 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.862112    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.915900 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.915684    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qfw\" (UniqueName: \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\") pod \"rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:31.924703 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:31.923536    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:32.302693 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:32.302667    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:21:32.303287 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:21:32.303263    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c71721_7dcf_469c_9a22_94c39622d17c.slice/crio-12640d901eb3f6ab3427b9e79515e5d8da54040fd6ab18c8a1dc060367148d0f WatchSource:0}: Error finding container 12640d901eb3f6ab3427b9e79515e5d8da54040fd6ab18c8a1dc060367148d0f: Status 404 returned error can't find the container with id 12640d901eb3f6ab3427b9e79515e5d8da54040fd6ab18c8a1dc060367148d0f
Nov 01 20:21:32.682098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:32.682002    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" event={"ID":"37c71721-7dcf-469c-9a22-94c39622d17c","Type":"ContainerStarted","Data":"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"}
Nov 01 20:21:32.682098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:32.682055    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" event={"ID":"37c71721-7dcf-469c-9a22-94c39622d17c","Type":"ContainerStarted","Data":"12640d901eb3f6ab3427b9e79515e5d8da54040fd6ab18c8a1dc060367148d0f"}
Nov 01 20:21:32.796372 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:32.796318    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" podStartSLOduration=1.7963035619999999 podStartE2EDuration="1.796303562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:32.794058082 +0000 UTC m=+1146.178955673" watchObservedRunningTime="2026-11-01 20:21:32.796303562 +0000 UTC m=+1146.181201153"
Nov 01 20:21:32.924095 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:32.924057    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:40.748271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.748231    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 20:21:40.752692 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.752671    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.809888 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.809854    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 20:21:40.853912 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.853882    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6a675a-serving-cert\""
Nov 01 20:21:40.875728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.875697    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.875877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.875835    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.875926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.875913    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.876009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.875986    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psff\" (UniqueName: \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.977445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.977403    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psff\" (UniqueName: \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.977659 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.977576    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.977731 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.977702    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.977792 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:40.977739    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-serving-cert: secret "maas-api-e2e-extmod-b-6a675a-serving-cert" not found
Nov 01 20:21:40.977848 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:40.977828    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls podName:55054cc1-2b4a-452e-84e7-ff27b6e9cc71 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:41.477801915 +0000 UTC m=+1154.862699488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls") pod "maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" (UID: "55054cc1-2b4a-452e-84e7-ff27b6e9cc71") : secret "maas-api-e2e-extmod-b-6a675a-serving-cert" not found
Nov 01 20:21:40.977913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.977866    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.978680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.978651    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:40.980303 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:40.980277    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:41.034869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.034798    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psff\" (UniqueName: \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:41.484070 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.484025    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:41.487030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.487000    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6a675a-767f8f57-prf62\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:41.668571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.668527    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:41.923779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.923672    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:41.938903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:41.938879    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:42.050334 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:42.050307    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 20:21:42.052111 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:21:42.052077    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55054cc1_2b4a_452e_84e7_ff27b6e9cc71.slice/crio-862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66 WatchSource:0}: Error finding container 862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66: Status 404 returned error can't find the container with id 862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66
Nov 01 20:21:42.761253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:42.761212    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" event={"ID":"55054cc1-2b4a-452e-84e7-ff27b6e9cc71","Type":"ContainerStarted","Data":"862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66"}
Nov 01 20:21:42.777841 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:42.777802    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:21:43.823018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:43.821231    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" event={"ID":"55054cc1-2b4a-452e-84e7-ff27b6e9cc71","Type":"ContainerStarted","Data":"efcbc8a91630a1ab120ed7909f7231732cc561868610021e5237f6c48c3c9df7"}
Nov 01 20:21:43.823018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:43.821570    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:43.856515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:43.856240    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:21:43.861283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:43.861249    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy" containerID="cri-o://2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9" gracePeriod=30
Nov 01 20:21:43.967284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:43.967207    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" podStartSLOduration=3.055473544 podStartE2EDuration="3.967188876s" totalImagesPullingTime="911.715332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:43.967028591 +0000 UTC m=+1157.351926183" watchObservedRunningTime="2026-11-01 20:21:43.967188876 +0000 UTC m=+1157.352086467"
Nov 01 20:21:49.146171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.146113    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:21:49.192433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192404    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192448    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192474    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192550    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192585    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192623    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192649    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192733    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192758    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\") pod \"551fc23f-2bca-44c2-9b06-2fd9cc626119\" (UID: \"551fc23f-2bca-44c2-9b06-2fd9cc626119\") "
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192759    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192846    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.192885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.192867    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.193276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.193253    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.193453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.193418    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.193710 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.193478    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.193710 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.193512    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.193710 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.193597    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:49.195048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.195023    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.195142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.195100    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "kube-api-access-q5h8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:49.195270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.195254    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:49.195270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.195257    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo" pod "551fc23f-2bca-44c2-9b06-2fd9cc626119" (UID: "551fc23f-2bca-44c2-9b06-2fd9cc626119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:21:49.295122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295091    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.295122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295118    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.295122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295129    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551fc23f-2bca-44c2-9b06-2fd9cc626119-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.295337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295140    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5h8b\" (UniqueName: \"kubernetes.io/projected/551fc23f-2bca-44c2-9b06-2fd9cc626119-kube-api-access-q5h8b\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.295337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295150    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.295337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.295159    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551fc23f-2bca-44c2-9b06-2fd9cc626119-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:21:49.837770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.837738    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 20:21:49.872570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.872537    5066 generic.go:356] "Generic (PLEG): container finished" podID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerID="2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9" exitCode=0
Nov 01 20:21:49.872763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.872603    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"
Nov 01 20:21:49.872763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.872625    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" event={"ID":"551fc23f-2bca-44c2-9b06-2fd9cc626119","Type":"ContainerDied","Data":"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"}
Nov 01 20:21:49.872763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.872684    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb" event={"ID":"551fc23f-2bca-44c2-9b06-2fd9cc626119","Type":"ContainerDied","Data":"4af4f26745dbc4902afc93c05a405a8966b5d9c09686e912695708594b798237"}
Nov 01 20:21:49.872763 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.872714    5066 scope.go:122] "RemoveContainer" containerID="2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"
Nov 01 20:21:49.885000 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.884981    5066 scope.go:122] "RemoveContainer" containerID="2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"
Nov 01 20:21:49.885265 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:49.885244    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9\": container with ID starting with 2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9 not found: ID does not exist" containerID="2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"
Nov 01 20:21:49.885335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:49.885273    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9"} err="failed to get container status \"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9\": rpc error: code = NotFound desc = could not find container \"2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9\": container with ID starting with 2f34c6b7fbde06b7070a82fa68e9a5fcc2301dce03c74c71caa4b7433f9f62f9 not found: ID does not exist"
Nov 01 20:21:50.069437 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:50.069406    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:21:50.073061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:50.073028    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-drkzb"]
Nov 01 20:21:50.732188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:21:50.732150    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" path="/var/lib/kubelet/pods/551fc23f-2bca-44c2-9b06-2fd9cc626119/volumes"
Nov 01 20:21:51.628971 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:21:51.628933    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:22:13.345512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:13.345455    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:22:18.346427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:18.346376    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing" containerID="cri-o://f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e" gracePeriod=25
Nov 01 20:22:21.629615 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:22:21.629571    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:22:24.987044 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:24.987000    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:22:26.897668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:26.897625    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:22:26.932786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:26.932764    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:22:34.987273 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:34.987236    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:22:38.634758 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.634733    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:22:38.837907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.837871    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\" (UniqueName: \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\") pod \"e559ba6a-9ff9-4219-997d-3c16639eea51\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") "
Nov 01 20:22:38.838073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.838028    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\") pod \"e559ba6a-9ff9-4219-997d-3c16639eea51\" (UID: \"e559ba6a-9ff9-4219-997d-3c16639eea51\") "
Nov 01 20:22:38.838389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.838366    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume" pod "e559ba6a-9ff9-4219-997d-3c16639eea51" (UID: "e559ba6a-9ff9-4219-997d-3c16639eea51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:22:38.838559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.838546    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e559ba6a-9ff9-4219-997d-3c16639eea51-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:22:38.840101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.840074    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf" pod "e559ba6a-9ff9-4219-997d-3c16639eea51" (UID: "e559ba6a-9ff9-4219-997d-3c16639eea51"). InnerVolumeSpecName "kube-api-access-cgzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:22:38.939259 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:38.939181    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgzrf\" (UniqueName: \"kubernetes.io/projected/e559ba6a-9ff9-4219-997d-3c16639eea51-kube-api-access-cgzrf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:22:39.247969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.247934    5066 generic.go:356] "Generic (PLEG): container finished" podID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerID="f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e" exitCode=0
Nov 01 20:22:39.248155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.247988    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"
Nov 01 20:22:39.248155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.248018    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" event={"ID":"e559ba6a-9ff9-4219-997d-3c16639eea51","Type":"ContainerDied","Data":"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"}
Nov 01 20:22:39.248155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.248069    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47" event={"ID":"e559ba6a-9ff9-4219-997d-3c16639eea51","Type":"ContainerDied","Data":"212b252fa2ee7438babc918f9306c2aa254f089161a5f4a4959ec1773e69000a"}
Nov 01 20:22:39.248155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.248091    5066 scope.go:122] "RemoveContainer" containerID="f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"
Nov 01 20:22:39.259008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.258991    5066 scope.go:122] "RemoveContainer" containerID="f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"
Nov 01 20:22:39.259310 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:22:39.259285    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e\": container with ID starting with f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e not found: ID does not exist" containerID="f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"
Nov 01 20:22:39.259386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.259322    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e"} err="failed to get container status \"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e\": rpc error: code = NotFound desc = could not find container \"f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e\": container with ID starting with f18eeeb819f5c9bb85614f083fccb3b7dc8012a36ebc02d416b457547526532e not found: ID does not exist"
Nov 01 20:22:39.414543 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.414508    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:22:39.465869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:39.465829    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ad7edb-64bdcdc987-h9f47"]
Nov 01 20:22:40.735034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:40.734988    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" path="/var/lib/kubelet/pods/e559ba6a-9ff9-4219-997d-3c16639eea51/volumes"
Nov 01 20:22:51.652373 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:22:51.652337    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:22:57.789164 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.789125    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 20:22:57.789974 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.789952    5066 cpu_manager.go:414] "containerMap: removing container" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing"
Nov 01 20:22:57.790078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.789975    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing"
Nov 01 20:22:57.790078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.790002    5066 cpu_manager.go:414] "containerMap: removing container" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy"
Nov 01 20:22:57.790078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.790011    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy"
Nov 01 20:22:57.790229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.790168    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="e559ba6a-9ff9-4219-997d-3c16639eea51" containerName="payload-processing"
Nov 01 20:22:57.790229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.790186    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="551fc23f-2bca-44c2-9b06-2fd9cc626119" containerName="istio-proxy"
Nov 01 20:22:57.794704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.794680    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.847023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.846982    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\""
Nov 01 20:22:57.902399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.902357    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 20:22:57.911732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.911696    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.911898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.911760    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.911898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.911864    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.911958    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcghk\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.912043    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.912091    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.912253    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.912300    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:57.912345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:57.912339    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013281    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013331    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcghk\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013375    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013402    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013527    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013558    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013592    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013728    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013742    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013811    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.013821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013811    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.014014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.013931    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.014049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.014035    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.014470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.014449    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.015863 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.015837    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.015972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.015905    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.069691 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.069628    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.069867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.069842    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcghk\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.116649 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.116619    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:22:58.564168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.564127    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 20:22:58.583067 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:22:58.582669    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaff215e_7392_40d2_b79e_9cb3227d92e2.slice/crio-016b6e2cedf2af71d002769cae29538d283b588058db3565e157b637379e7e32 WatchSource:0}: Error finding container 016b6e2cedf2af71d002769cae29538d283b588058db3565e157b637379e7e32: Status 404 returned error can't find the container with id 016b6e2cedf2af71d002769cae29538d283b588058db3565e157b637379e7e32
Nov 01 20:22:58.585624 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.585586    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:22:58.585737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.585673    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:22:58.585737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:58.585716    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:22:59.484516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:59.483699    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" event={"ID":"caff215e-7392-40d2-b79e-9cb3227d92e2","Type":"ContainerStarted","Data":"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"}
Nov 01 20:22:59.484516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:59.483769    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" event={"ID":"caff215e-7392-40d2-b79e-9cb3227d92e2","Type":"ContainerStarted","Data":"016b6e2cedf2af71d002769cae29538d283b588058db3565e157b637379e7e32"}
Nov 01 20:22:59.706095 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:22:59.706022    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podStartSLOduration=2.706002083 podStartE2EDuration="2.706002083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:59.654591607 +0000 UTC m=+1233.039489200" watchObservedRunningTime="2026-11-01 20:22:59.706002083 +0000 UTC m=+1233.090899677"
Nov 01 20:23:00.117153 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:00.117111    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:23:00.119112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:00.119073    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 20:23:00.119263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:00.119136    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 20:23:01.120518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.119933    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 20:23:01.120518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.120009    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 20:23:01.387592 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.363672    5066 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podUID="2171233f-acc1-4ada-84ad-bfdfa7d4c834" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:23:01.561515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.540716    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:23:01.603520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.592524    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" containerID="cri-o://6b6b44d5bcbf834cde1c14de7dea4c7873ad15f5ba143052dd5e186de1fe78d4" gracePeriod=30
Nov 01 20:23:01.675286 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:01.675179    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podUID="2171233f-acc1-4ada-84ad-bfdfa7d4c834" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:23:02.118553 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:02.118461    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 20:23:02.118553 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:02.118544    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 20:23:03.117202 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:03.117156    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 20:23:03.117732 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:03.117232    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 20:23:04.117332 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:04.117295    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 20:23:04.117768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:04.117366    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 20:23:04.841951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:04.841920    5066 patch_prober.go:29] interesting pod/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:23:04.842128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:04.841979    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:23:05.223089 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:05.223051    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:23:05.223686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:05.223659    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:23:05.224800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:05.224780    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 20:23:06.842238 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:06.842205    5066 generic.go:356] "Generic (PLEG): container finished" podID="d4a661cb-d33b-492a-b991-99940793995b" containerID="6b6b44d5bcbf834cde1c14de7dea4c7873ad15f5ba143052dd5e186de1fe78d4" exitCode=0
Nov 01 20:23:06.842704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:06.842283    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" event={"ID":"d4a661cb-d33b-492a-b991-99940793995b","Type":"ContainerDied","Data":"6b6b44d5bcbf834cde1c14de7dea4c7873ad15f5ba143052dd5e186de1fe78d4"}
Nov 01 20:23:06.962593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:06.962508    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:23:07.139873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.139838    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.139928    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140026    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140065    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140105    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140125    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:07.140257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140237    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140269    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140300    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140379    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\") pod \"d4a661cb-d33b-492a-b991-99940793995b\" (UID: \"d4a661cb-d33b-492a-b991-99940793995b\") "
Nov 01 20:23:07.140628 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140564    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:07.140745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140726    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:07.140990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.140971    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.141132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.141116    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.141230 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.141216    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.141304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.141248    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:07.141379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.141356    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:07.143129 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.143097    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:07.143240 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.143151    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "kube-api-access-pgnnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:07.143653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.143622    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:23:07.144182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.144156    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token" pod "d4a661cb-d33b-492a-b991-99940793995b" (UID: "d4a661cb-d33b-492a-b991-99940793995b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:07.242857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242825    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4a661cb-d33b-492a-b991-99940793995b-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.242857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242857    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.243042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242867    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.243042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242876    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4a661cb-d33b-492a-b991-99940793995b-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.243042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242884    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4a661cb-d33b-492a-b991-99940793995b-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.243042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.242894    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgnnc\" (UniqueName: \"kubernetes.io/projected/d4a661cb-d33b-492a-b991-99940793995b-kube-api-access-pgnnc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.851293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.851255    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x" event={"ID":"d4a661cb-d33b-492a-b991-99940793995b","Type":"ContainerDied","Data":"9dd716bb1f024c9a82975065d1ef7109e9d1d42b45ae741e054b6dcefeab53c3"}
Nov 01 20:23:07.851776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.851327    5066 scope.go:122] "RemoveContainer" containerID="6b6b44d5bcbf834cde1c14de7dea4c7873ad15f5ba143052dd5e186de1fe78d4"
Nov 01 20:23:07.851776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:07.851259    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"
Nov 01 20:23:08.069016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:08.068979    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:23:08.118514 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:08.118412    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ad7edb-openshift-default-8445b9677-cxx7x"]
Nov 01 20:23:08.732013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:08.731974    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a661cb-d33b-492a-b991-99940793995b" path="/var/lib/kubelet/pods/d4a661cb-d33b-492a-b991-99940793995b/volumes"
Nov 01 20:23:08.919676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:08.919635    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:23:09.029057 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:09.028971    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:23:13.920631 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:13.920590    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing" containerID="cri-o://54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832" gracePeriod=25
Nov 01 20:23:14.030086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:14.030026    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing" containerID="cri-o://421a4a438a83cde31573df6f195b37dd8a0d1c46c6351a153d8f390669e25b6d" gracePeriod=25
Nov 01 20:23:21.602375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:21.602337    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:23:21.602375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:21.602358    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:23:21.629991 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:23:21.629950    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:23:31.600676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:31.600632    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:23:31.601220 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:31.600647    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:23:34.208227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.208198    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:23:34.293170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.293140    5066 generic.go:356] "Generic (PLEG): container finished" podID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerID="54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832" exitCode=0
Nov 01 20:23:34.293307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.293187    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"
Nov 01 20:23:34.293307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.293216    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" event={"ID":"b92bca7b-a3d4-4838-a533-37f93c5897a7","Type":"ContainerDied","Data":"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"}
Nov 01 20:23:34.293307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.293282    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf" event={"ID":"b92bca7b-a3d4-4838-a533-37f93c5897a7","Type":"ContainerDied","Data":"345e9389f95cdff394a98ed222a6e4fcded0a8f16657010a98b8b4a1bc350f0e"}
Nov 01 20:23:34.293469 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.293313    5066 scope.go:122] "RemoveContainer" containerID="54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"
Nov 01 20:23:34.294706 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.294687    5066 generic.go:356] "Generic (PLEG): container finished" podID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerID="421a4a438a83cde31573df6f195b37dd8a0d1c46c6351a153d8f390669e25b6d" exitCode=0
Nov 01 20:23:34.294812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.294760    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" event={"ID":"975a4985-b1d8-4673-8bea-1a5ac81aff57","Type":"ContainerDied","Data":"421a4a438a83cde31573df6f195b37dd8a0d1c46c6351a153d8f390669e25b6d"}
Nov 01 20:23:34.294875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.294833    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx" event={"ID":"975a4985-b1d8-4673-8bea-1a5ac81aff57","Type":"ContainerDied","Data":"e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380"}
Nov 01 20:23:34.294875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.294852    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77ce02da88f63d136489e7fc24cd004482a1769ef2d0acab10f630f75f8e380"
Nov 01 20:23:34.304094 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.304063    5066 scope.go:122] "RemoveContainer" containerID="54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"
Nov 01 20:23:34.304425 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:23:34.304404    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832\": container with ID starting with 54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832 not found: ID does not exist" containerID="54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"
Nov 01 20:23:34.304551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.304432    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832"} err="failed to get container status \"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832\": rpc error: code = NotFound desc = could not find container \"54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832\": container with ID starting with 54befa9d82d363ab377fe9df7fc6dfc856f328ab10c970fdbdd9ee3dd714d832 not found: ID does not exist"
Nov 01 20:23:34.304551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.304527    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:23:34.324882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.324856    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\") pod \"b92bca7b-a3d4-4838-a533-37f93c5897a7\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") "
Nov 01 20:23:34.325010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.324911    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\" (UniqueName: \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\") pod \"b92bca7b-a3d4-4838-a533-37f93c5897a7\" (UID: \"b92bca7b-a3d4-4838-a533-37f93c5897a7\") "
Nov 01 20:23:34.325280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.325253    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume" pod "b92bca7b-a3d4-4838-a533-37f93c5897a7" (UID: "b92bca7b-a3d4-4838-a533-37f93c5897a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:34.325435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.325417    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92bca7b-a3d4-4838-a533-37f93c5897a7-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:34.327227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.327199    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x" pod "b92bca7b-a3d4-4838-a533-37f93c5897a7" (UID: "b92bca7b-a3d4-4838-a533-37f93c5897a7"). InnerVolumeSpecName "kube-api-access-n4g7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:34.426048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.426015    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\" (UniqueName: \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\") pod \"975a4985-b1d8-4673-8bea-1a5ac81aff57\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") "
Nov 01 20:23:34.426223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.426065    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\") pod \"975a4985-b1d8-4673-8bea-1a5ac81aff57\" (UID: \"975a4985-b1d8-4673-8bea-1a5ac81aff57\") "
Nov 01 20:23:34.426439 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.426415    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume" pod "975a4985-b1d8-4673-8bea-1a5ac81aff57" (UID: "975a4985-b1d8-4673-8bea-1a5ac81aff57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:34.426638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.426623    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4g7x\" (UniqueName: \"kubernetes.io/projected/b92bca7b-a3d4-4838-a533-37f93c5897a7-kube-api-access-n4g7x\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:34.426685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.426640    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975a4985-b1d8-4673-8bea-1a5ac81aff57-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:34.428103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.428084    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj" pod "975a4985-b1d8-4673-8bea-1a5ac81aff57" (UID: "975a4985-b1d8-4673-8bea-1a5ac81aff57"). InnerVolumeSpecName "kube-api-access-7dxzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:34.528389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.528350    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dxzj\" (UniqueName: \"kubernetes.io/projected/975a4985-b1d8-4673-8bea-1a5ac81aff57-kube-api-access-7dxzj\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:34.716700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.716665    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:23:34.719601 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.719577    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d5118-5c9d95b959-pscsf"]
Nov 01 20:23:34.731891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:34.731859    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" path="/var/lib/kubelet/pods/b92bca7b-a3d4-4838-a533-37f93c5897a7/volumes"
Nov 01 20:23:35.302395 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:35.302368    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"
Nov 01 20:23:35.466444 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:35.466402    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:23:35.469377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:35.469356    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7d5118-5494cdc6f5-gvbhx"]
Nov 01 20:23:36.731695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:36.731661    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" path="/var/lib/kubelet/pods/975a4985-b1d8-4673-8bea-1a5ac81aff57/volumes"
Nov 01 20:23:41.206459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.206425    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:23:41.206921 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.206640    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" podUID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" containerName="maas-api" containerID="cri-o://4b4471278ca570ffad0e50d71f93be0679d4e117bc3e8a04ec245308367aea67" gracePeriod=30
Nov 01 20:23:41.352575 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.352544    5066 generic.go:356] "Generic (PLEG): container finished" podID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" containerID="4b4471278ca570ffad0e50d71f93be0679d4e117bc3e8a04ec245308367aea67" exitCode=0
Nov 01 20:23:41.352739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.352624    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" event={"ID":"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5","Type":"ContainerDied","Data":"4b4471278ca570ffad0e50d71f93be0679d4e117bc3e8a04ec245308367aea67"}
Nov 01 20:23:41.566825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.566796    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:23:41.737094 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737058    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\" (UniqueName: \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\") pod \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") "
Nov 01 20:23:41.737267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737231    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\") pod \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") "
Nov 01 20:23:41.737363 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737318    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") pod \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") "
Nov 01 20:23:41.737411 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737363    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\") pod \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\" (UID: \"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5\") "
Nov 01 20:23:41.737795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737765    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca" pod "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" (UID: "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:41.737920 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.737898    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:41.739380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.739355    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls" pod "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" (UID: "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:41.739766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.739747    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs" pod "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" (UID: "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:41.739828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.739788    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr" pod "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" (UID: "f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5"). InnerVolumeSpecName "kube-api-access-qkpfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:41.839371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.839289    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:41.839371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.839318    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:41.839371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:41.839328    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpfr\" (UniqueName: \"kubernetes.io/projected/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5-kube-api-access-qkpfr\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:42.359987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.359957    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"
Nov 01 20:23:42.360404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.360008    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp" event={"ID":"f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5","Type":"ContainerDied","Data":"fc05038ce978532a4df96ceab1948010383c7c078809aa1f4554c5be612b801d"}
Nov 01 20:23:42.360404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.360072    5066 scope.go:122] "RemoveContainer" containerID="4b4471278ca570ffad0e50d71f93be0679d4e117bc3e8a04ec245308367aea67"
Nov 01 20:23:42.522241 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.522206    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:23:42.571896 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.571861    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d5118-79f8d96845-lq9bp"]
Nov 01 20:23:42.731389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:42.731359    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" path="/var/lib/kubelet/pods/f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5/volumes"
Nov 01 20:23:46.521082 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:46.521047    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:23:46.521485 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:46.521264    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy" containerID="cri-o://7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104" gracePeriod=30
Nov 01 20:23:51.630454 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:23:51.630412    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:23:51.808109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.808079    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:23:51.886794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.886762    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.886999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.886805    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.886999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.886844    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.886999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.886887    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.886999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.886994    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.887210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887022    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.887210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887122    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:51.887210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887167    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.887210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887201    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:51.887413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887296    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.887413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887359    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\") pod \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\" (UID: \"d2906cec-f4c8-4191-b39f-1758f1adb4e0\") "
Nov 01 20:23:51.887653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887462    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:51.887948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887926    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.888053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887956    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.888053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.887972    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.888447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.888410    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:51.888777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.888635    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:51.889873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.889849    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:23:51.890303 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.890277    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:51.890420 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.890396    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:51.890619 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.890434    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb" pod "d2906cec-f4c8-4191-b39f-1758f1adb4e0" (UID: "d2906cec-f4c8-4191-b39f-1758f1adb4e0"). InnerVolumeSpecName "kube-api-access-kmsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:51.991386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.991109    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.991386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.991147    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.991386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.991161    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.991386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.991175    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmsqb\" (UniqueName: \"kubernetes.io/projected/d2906cec-f4c8-4191-b39f-1758f1adb4e0-kube-api-access-kmsqb\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.991386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.991190    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:51.993992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:51.993957    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2906cec-f4c8-4191-b39f-1758f1adb4e0-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:23:52.425841 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.425809    5066 generic.go:356] "Generic (PLEG): container finished" podID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerID="7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104" exitCode=0
Nov 01 20:23:52.426023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.425858    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"
Nov 01 20:23:52.426023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.425885    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerDied","Data":"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"}
Nov 01 20:23:52.426023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.425935    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6" event={"ID":"d2906cec-f4c8-4191-b39f-1758f1adb4e0","Type":"ContainerDied","Data":"049c69c303a77a7a3aa87b8c62cfd249316c0815bfe72b82990520b0e7573338"}
Nov 01 20:23:52.426023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.425959    5066 scope.go:122] "RemoveContainer" containerID="7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"
Nov 01 20:23:52.437195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.437180    5066 scope.go:122] "RemoveContainer" containerID="5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"
Nov 01 20:23:52.445981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.445965    5066 scope.go:122] "RemoveContainer" containerID="7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"
Nov 01 20:23:52.446245 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:23:52.446227    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104\": container with ID starting with 7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104 not found: ID does not exist" containerID="7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"
Nov 01 20:23:52.446292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.446253    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104"} err="failed to get container status \"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104\": rpc error: code = NotFound desc = could not find container \"7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104\": container with ID starting with 7ef217820fb9a4388cda4a1750197af2c7522614eb36f0a2ea1fa360985f0104 not found: ID does not exist"
Nov 01 20:23:52.446292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.446270    5066 scope.go:122] "RemoveContainer" containerID="5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"
Nov 01 20:23:52.446539 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:23:52.446479    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff\": container with ID starting with 5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff not found: ID does not exist" containerID="5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"
Nov 01 20:23:52.446630 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.446537    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff"} err="failed to get container status \"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff\": rpc error: code = NotFound desc = could not find container \"5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff\": container with ID starting with 5dbf8acf3fc68137adf10d3829e7ec33d3d5f37d2f0e30c9083787108aaaacff not found: ID does not exist"
Nov 01 20:23:52.592189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.592151    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:23:52.639785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.639752    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-jh9p6"]
Nov 01 20:23:52.734588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:52.734546    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" path="/var/lib/kubelet/pods/d2906cec-f4c8-4191-b39f-1758f1adb4e0/volumes"
Nov 01 20:23:56.589026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.588982    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:23:56.589652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589635    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589652    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589663    5066 cpu_manager.go:414] "containerMap: removing container" podUID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" containerName="maas-api"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589668    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" containerName="maas-api"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589690    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589695    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589712    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589717    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing"
Nov 01 20:23:56.589733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589734    5066 cpu_manager.go:414] "containerMap: removing container" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589739    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589749    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589754    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589827    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="975a4985-b1d8-4673-8bea-1a5ac81aff57" containerName="payload-pre-processing"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589835    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a661cb-d33b-492a-b991-99940793995b" containerName="istio-proxy"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589842    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589848    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b0c56e-38b4-4d93-a66f-5f66b2e36ec5" containerName="maas-api"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589857    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2906cec-f4c8-4191-b39f-1758f1adb4e0" containerName="istio-proxy"
Nov 01 20:23:56.589964 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.589865    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92bca7b-a3d4-4838-a533-37f93c5897a7" containerName="payload-processing"
Nov 01 20:23:56.594780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.594761    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.646580 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.646538    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2a4e4430-openshift-default-dockercfg-47kj5\""
Nov 01 20:23:56.701538 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.701480    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.751450    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.751541    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.751622    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.751670    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.756748    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pnzz\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.756831    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.756881    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.756923    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.757515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.757086    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858009    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858056    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858168    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858198    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858251    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858281    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858309    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pnzz\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858336    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858368    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858631    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858726    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858778    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.858840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.858814    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.859027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.859008    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.860711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.860685    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.861023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.861001    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.915894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.915858    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:56.916076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:56.915976    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pnzz\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\") pod \"e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") " pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:57.212681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:57.212589    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:57.642403 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:57.642376    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:23:57.643237 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:23:57.643213    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe1a3a3_f0fc_407c_b43d_3621b7e32706.slice/crio-af299f8be3824fad5781297aa8ded39debe6356d984e287583ebc3263bb3b921 WatchSource:0}: Error finding container af299f8be3824fad5781297aa8ded39debe6356d984e287583ebc3263bb3b921: Status 404 returned error can't find the container with id af299f8be3824fad5781297aa8ded39debe6356d984e287583ebc3263bb3b921
Nov 01 20:23:57.645840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:57.645802    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:23:57.645951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:57.645866    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:23:57.645951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:57.645899    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:23:58.502577 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:58.502541    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerStarted","Data":"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"}
Nov 01 20:23:58.502836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:58.502590    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerStarted","Data":"af299f8be3824fad5781297aa8ded39debe6356d984e287583ebc3263bb3b921"}
Nov 01 20:23:58.666077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:58.666018    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podStartSLOduration=2.666004779 podStartE2EDuration="2.666004779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:58.663048036 +0000 UTC m=+1292.047945628" watchObservedRunningTime="2026-11-01 20:23:58.666004779 +0000 UTC m=+1292.050902370"
Nov 01 20:23:59.212713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:59.212674    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:59.218252 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:59.218226    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:59.512774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:59.512738    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:23:59.518622 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:23:59.518587    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:24:06.408925 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.408888    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:24:06.413310 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.413282    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.513253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.513215    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.513430 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.513332    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xqd\" (UniqueName: \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.514847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.514823    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2a4e4430-dockercfg-qpf72\""
Nov 01 20:24:06.514989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.514900    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2a4e4430\""
Nov 01 20:24:06.518094 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.518073    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:24:06.614702 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.614670    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.614905 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.614740    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xqd\" (UniqueName: \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.615386 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.615367    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.671209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.671140    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xqd\" (UniqueName: \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\") pod \"payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:06.728444 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:06.728419    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:07.101394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:07.101367    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:24:07.102894 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:24:07.102873    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145778b2_476f_40e2_90c5_1ffa439a1adf.slice/crio-01768eb6b1f05ea96bcd2ba45a39161dd6f064a874448b45b15f63baf09af9d6 WatchSource:0}: Error finding container 01768eb6b1f05ea96bcd2ba45a39161dd6f064a874448b45b15f63baf09af9d6: Status 404 returned error can't find the container with id 01768eb6b1f05ea96bcd2ba45a39161dd6f064a874448b45b15f63baf09af9d6
Nov 01 20:24:07.567623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:07.567592    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" event={"ID":"145778b2-476f-40e2-90c5-1ffa439a1adf","Type":"ContainerStarted","Data":"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"}
Nov 01 20:24:07.568078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:07.567647    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" event={"ID":"145778b2-476f-40e2-90c5-1ffa439a1adf","Type":"ContainerStarted","Data":"01768eb6b1f05ea96bcd2ba45a39161dd6f064a874448b45b15f63baf09af9d6"}
Nov 01 20:24:07.568078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:07.567675    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:07.681047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:07.680991    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" podStartSLOduration=1.680974284 podStartE2EDuration="1.680974284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:07.678789136 +0000 UTC m=+1301.063686727" watchObservedRunningTime="2026-11-01 20:24:07.680974284 +0000 UTC m=+1301.065871876"
Nov 01 20:24:18.578635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:18.578598    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:24:21.628750 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:24:21.628713    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:24:27.086203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:27.086149    5066 scope.go:122] "RemoveContainer" containerID="421a4a438a83cde31573df6f195b37dd8a0d1c46c6351a153d8f390669e25b6d"
Nov 01 20:24:37.172573 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.172536    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:24:37.178075 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.178056    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.250515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.233912    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:24:37.293521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.282748    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2a4e4430-serving-cert\""
Nov 01 20:24:37.309890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.294420    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.328266 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.328232    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jhn\" (UniqueName: \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.328466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.328348    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.328557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.328505    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.429948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.429858    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.429948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.429922    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.430181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.430036    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jhn\" (UniqueName: \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.430181 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:24:37.430057    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-serving-cert: secret "maas-api-e2e-ait-2a4e4430-serving-cert" not found
Nov 01 20:24:37.430181 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:24:37.430140    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls podName:3f917b21-d9c9-4a29-8836-8894a6fb21a5 nodeName:}" failed. No retries permitted until 2026-11-01 20:24:37.930107806 +0000 UTC m=+1331.315005391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls") pod "maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" (UID: "3f917b21-d9c9-4a29-8836-8894a6fb21a5") : secret "maas-api-e2e-ait-2a4e4430-serving-cert" not found
Nov 01 20:24:37.430611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.430183    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.431138 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.431114    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.433340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.433313    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.487370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.487332    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jhn\" (UniqueName: \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.950295 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.950253    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:37.954062 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:37.953964    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") pod \"maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:38.093757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:38.093722    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:39.260841 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.248454    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:24:39.365826 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:24:39.364669    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f917b21_d9c9_4a29_8836_8894a6fb21a5.slice/crio-f15d52a4e98703c5c09d6f5cf0fcc35dad8d678a5f86845cd9d20eda5fea5640 WatchSource:0}: Error finding container f15d52a4e98703c5c09d6f5cf0fcc35dad8d678a5f86845cd9d20eda5fea5640: Status 404 returned error can't find the container with id f15d52a4e98703c5c09d6f5cf0fcc35dad8d678a5f86845cd9d20eda5fea5640
Nov 01 20:24:39.529015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.528615    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:24:39.529015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.528684    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:39.592814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.592646    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:24:39.592814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.592731    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:39.629296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.627980    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:39.728167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.727794    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podUID="2171233f-acc1-4ada-84ad-bfdfa7d4c834" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:39.787646 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.787544    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Nov 01 20:24:39.794765 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.794640    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:24:39.794765 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:39.794702    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:40.245119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:40.245078    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" event={"ID":"3f917b21-d9c9-4a29-8836-8894a6fb21a5","Type":"ContainerStarted","Data":"f15d52a4e98703c5c09d6f5cf0fcc35dad8d678a5f86845cd9d20eda5fea5640"}
Nov 01 20:24:41.254454 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:41.254418    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" event={"ID":"3f917b21-d9c9-4a29-8836-8894a6fb21a5","Type":"ContainerStarted","Data":"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"}
Nov 01 20:24:41.254949 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:41.254685    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:41.369731 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:41.369671    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" podStartSLOduration=3.321810528 podStartE2EDuration="4.369656256s" totalImagesPullingTime="1.047845728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:41.367748195 +0000 UTC m=+1334.752645800" watchObservedRunningTime="2026-11-01 20:24:41.369656256 +0000 UTC m=+1334.754553846"
Nov 01 20:24:47.268941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:47.268903    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:24:51.628768 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:24:51.628729    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:24:56.548660 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:56.548615    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:56.668628 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:24:56.668577    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv" podUID="2171233f-acc1-4ada-84ad-bfdfa7d4c834" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:05.491067 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.491029    5066 generic.go:356] "Generic (PLEG): container finished" podID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerID="dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb" exitCode=0
Nov 01 20:25:05.491627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.491115    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerDied","Data":"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"}
Nov 01 20:25:05.491627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.491614    5066 scope.go:122] "RemoveContainer" containerID="dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"
Nov 01 20:25:05.493272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.493220    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:25:05.493366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.493326    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:25:05.493404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:05.493375    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:25:06.500881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:06.500841    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerStarted","Data":"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"}
Nov 01 20:25:07.212997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:07.212963    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:25:07.213214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:07.213158    5066 patch_prober.go:29] interesting pod/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 20:25:07.213258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:07.213213    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 20:25:08.212933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:08.212896    5066 patch_prober.go:29] interesting pod/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 20:25:08.213416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:08.212950    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 20:25:08.992335 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:08.992291    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:25:08.997042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:08.997018    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.040262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.040223    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:25:09.049152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.049121    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.049474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.049456    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77kn\" (UniqueName: \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.128265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.128233    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71335882-dockercfg-s9mtf\""
Nov 01 20:25:09.128626 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.128605    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71335882\""
Nov 01 20:25:09.176316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.176272    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77kn\" (UniqueName: \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.176740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.176716    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.215649 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.187429    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.217663 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.217620    5066 patch_prober.go:29] interesting pod/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 20:25:09.217811 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.217702    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 20:25:09.224585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.224549    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:25:09.231734 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.231703    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:25:09.231877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.231821    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.245089 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.244696    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77kn\" (UniqueName: \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\") pod \"payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.310931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.310894    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:09.317845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.317814    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.318180 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.318159    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8xg\" (UniqueName: \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.423610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.423041    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8xg\" (UniqueName: \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.423610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.423246    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.427169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.424294    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.520821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.520727    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8xg\" (UniqueName: \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\") pod \"payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") " pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.549626 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.549489    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:09.726840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.726453    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:25:09.999882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:09.999850    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:25:10.213655 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.213618    5066 patch_prober.go:29] interesting pod/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 20:25:10.213823 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.213679    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 20:25:10.542725 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.542691    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" event={"ID":"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d","Type":"ContainerStarted","Data":"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"}
Nov 01 20:25:10.543228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.542741    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" event={"ID":"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d","Type":"ContainerStarted","Data":"5c130102c2fe1cc8598fcc4c49176aaef55fb22151f291d333ccc6b2674ce5c7"}
Nov 01 20:25:10.543228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.542904    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:10.544486 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.544460    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" event={"ID":"524a98e1-69f6-4871-b339-49cfc30612e3","Type":"ContainerStarted","Data":"81dac954220e15a1a327db8fdbb0bf1cda5d468c48f1096eb704cabb70f49d00"}
Nov 01 20:25:10.544613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.544526    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" event={"ID":"524a98e1-69f6-4871-b339-49cfc30612e3","Type":"ContainerStarted","Data":"406af13af003f5668939d630451d58494846eaa8217e7a82c0d9979490708339"}
Nov 01 20:25:10.544613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.544552    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:10.656480 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.656415    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" podStartSLOduration=2.656395541 podStartE2EDuration="2.656395541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:10.65425371 +0000 UTC m=+1364.039151301" watchObservedRunningTime="2026-11-01 20:25:10.656395541 +0000 UTC m=+1364.041293133"
Nov 01 20:25:10.805296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:10.799989    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" podStartSLOduration=1.799967493 podStartE2EDuration="1.799967493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:10.775882333 +0000 UTC m=+1364.160779925" watchObservedRunningTime="2026-11-01 20:25:10.799967493 +0000 UTC m=+1364.184865086"
Nov 01 20:25:11.213563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:11.213117    5066 patch_prober.go:29] interesting pod/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 20:25:11.213563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:11.213171    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 20:25:12.217802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:12.217768    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:25:12.218290 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:12.218164    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:25:12.218825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:12.218805    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:25:21.554470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:21.554437    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:25:21.554898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:21.554654    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:25:21.630542 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:25:21.630503    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:25:39.845738 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:39.845649    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:25:44.846303 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:44.846257    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" podUID="145778b2-476f-40e2-90c5-1ffa439a1adf" containerName="payload-processing" containerID="cri-o://1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7" gracePeriod=25
Nov 01 20:25:45.132302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.132277    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:25:45.164632 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.164596    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\") pod \"145778b2-476f-40e2-90c5-1ffa439a1adf\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") "
Nov 01 20:25:45.164795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.164687    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\" (UniqueName: \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\") pod \"145778b2-476f-40e2-90c5-1ffa439a1adf\" (UID: \"145778b2-476f-40e2-90c5-1ffa439a1adf\") "
Nov 01 20:25:45.164985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.164957    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume" pod "145778b2-476f-40e2-90c5-1ffa439a1adf" (UID: "145778b2-476f-40e2-90c5-1ffa439a1adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:45.165283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.165268    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145778b2-476f-40e2-90c5-1ffa439a1adf-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:25:45.166875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.166851    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd" pod "145778b2-476f-40e2-90c5-1ffa439a1adf" (UID: "145778b2-476f-40e2-90c5-1ffa439a1adf"). InnerVolumeSpecName "kube-api-access-c7xqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:45.266743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.266705    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xqd\" (UniqueName: \"kubernetes.io/projected/145778b2-476f-40e2-90c5-1ffa439a1adf-kube-api-access-c7xqd\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:25:45.928560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.928519    5066 generic.go:356] "Generic (PLEG): container finished" podID="145778b2-476f-40e2-90c5-1ffa439a1adf" containerID="1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7" exitCode=0
Nov 01 20:25:45.928995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.928562    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"
Nov 01 20:25:45.928995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.928600    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" event={"ID":"145778b2-476f-40e2-90c5-1ffa439a1adf","Type":"ContainerDied","Data":"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"}
Nov 01 20:25:45.928995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.928657    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27" event={"ID":"145778b2-476f-40e2-90c5-1ffa439a1adf","Type":"ContainerDied","Data":"01768eb6b1f05ea96bcd2ba45a39161dd6f064a874448b45b15f63baf09af9d6"}
Nov 01 20:25:45.928995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.928679    5066 scope.go:122] "RemoveContainer" containerID="1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"
Nov 01 20:25:45.939781 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.939757    5066 scope.go:122] "RemoveContainer" containerID="1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"
Nov 01 20:25:45.940078 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:25:45.940055    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7\": container with ID starting with 1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7 not found: ID does not exist" containerID="1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"
Nov 01 20:25:45.940164 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:45.940087    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7"} err="failed to get container status \"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7\": rpc error: code = NotFound desc = could not find container \"1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7\": container with ID starting with 1ba5b88265a2d25356dfd2334543f330c2e9b363c5d92b8702116d92b203f1b7 not found: ID does not exist"
Nov 01 20:25:46.100571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:46.100528    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:25:46.103648 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:46.103623    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4e4430-5dbd666fd9-4zm27"]
Nov 01 20:25:46.732020 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:25:46.731974    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145778b2-476f-40e2-90c5-1ffa439a1adf" path="/var/lib/kubelet/pods/145778b2-476f-40e2-90c5-1ffa439a1adf/volumes"
Nov 01 20:25:51.630689 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:25:51.630642    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:26:03.322430 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.322388    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:26:03.322867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.322650    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" podUID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" containerName="maas-api" containerID="cri-o://09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025" gracePeriod=30
Nov 01 20:26:03.632369 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.632334    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:26:03.723745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.723710    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\") pod \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") "
Nov 01 20:26:03.723951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.723902    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\" (UniqueName: \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\") pod \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") "
Nov 01 20:26:03.724046 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.724029    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\") pod \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") "
Nov 01 20:26:03.724110 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.724070    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca" pod "3f917b21-d9c9-4a29-8836-8894a6fb21a5" (UID: "3f917b21-d9c9-4a29-8836-8894a6fb21a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:03.724171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.724104    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") pod \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\" (UID: \"3f917b21-d9c9-4a29-8836-8894a6fb21a5\") "
Nov 01 20:26:03.724475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.724459    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f917b21-d9c9-4a29-8836-8894a6fb21a5-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:03.726097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.726073    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs" pod "3f917b21-d9c9-4a29-8836-8894a6fb21a5" (UID: "3f917b21-d9c9-4a29-8836-8894a6fb21a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:03.726183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.726161    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn" pod "3f917b21-d9c9-4a29-8836-8894a6fb21a5" (UID: "3f917b21-d9c9-4a29-8836-8894a6fb21a5"). InnerVolumeSpecName "kube-api-access-k5jhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:03.726223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.726181    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls" pod "3f917b21-d9c9-4a29-8836-8894a6fb21a5" (UID: "3f917b21-d9c9-4a29-8836-8894a6fb21a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:03.825611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.825576    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jhn\" (UniqueName: \"kubernetes.io/projected/3f917b21-d9c9-4a29-8836-8894a6fb21a5-kube-api-access-k5jhn\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:03.825611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.825608    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:03.825611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:03.825618    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f917b21-d9c9-4a29-8836-8894a6fb21a5-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:04.098757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.098715    5066 generic.go:356] "Generic (PLEG): container finished" podID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" containerID="09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025" exitCode=0
Nov 01 20:26:04.098954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.098775    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"
Nov 01 20:26:04.098954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.098892    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" event={"ID":"3f917b21-d9c9-4a29-8836-8894a6fb21a5","Type":"ContainerDied","Data":"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"}
Nov 01 20:26:04.098954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.098938    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv" event={"ID":"3f917b21-d9c9-4a29-8836-8894a6fb21a5","Type":"ContainerDied","Data":"f15d52a4e98703c5c09d6f5cf0fcc35dad8d678a5f86845cd9d20eda5fea5640"}
Nov 01 20:26:04.099152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.098968    5066 scope.go:122] "RemoveContainer" containerID="09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"
Nov 01 20:26:04.161919 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.161547    5066 scope.go:122] "RemoveContainer" containerID="09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"
Nov 01 20:26:04.162277 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:26:04.162171    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025\": container with ID starting with 09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025 not found: ID does not exist" containerID="09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"
Nov 01 20:26:04.162277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.162215    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025"} err="failed to get container status \"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025\": rpc error: code = NotFound desc = could not find container \"09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025\": container with ID starting with 09065e2d70e86468354e7c0a415a0a13e498b404c0e2c469951761b1dc8a3025 not found: ID does not exist"
Nov 01 20:26:04.264356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.264318    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:26:04.267866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.267838    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4e4430-7d46778dfd-nzbxv"]
Nov 01 20:26:04.731945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:04.731902    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" path="/var/lib/kubelet/pods/3f917b21-d9c9-4a29-8836-8894a6fb21a5/volumes"
Nov 01 20:26:05.568047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:05.568015    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:26:05.568253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:05.568232    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy" containerID="cri-o://30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501" gracePeriod=30
Nov 01 20:26:07.035954 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.035850    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:26:07.037635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.037608    5066 cpu_manager.go:414] "containerMap: removing container" podUID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" containerName="maas-api"
Nov 01 20:26:07.037789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.037767    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" containerName="maas-api"
Nov 01 20:26:07.037932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.037829    5066 cpu_manager.go:414] "containerMap: removing container" podUID="145778b2-476f-40e2-90c5-1ffa439a1adf" containerName="payload-processing"
Nov 01 20:26:07.037932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.037838    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145778b2-476f-40e2-90c5-1ffa439a1adf" containerName="payload-processing"
Nov 01 20:26:07.038046 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.037994    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f917b21-d9c9-4a29-8836-8894a6fb21a5" containerName="maas-api"
Nov 01 20:26:07.038046 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.038011    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="145778b2-476f-40e2-90c5-1ffa439a1adf" containerName="payload-processing"
Nov 01 20:26:07.043594 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.043569    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.096137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.095826    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-46c731c3-gw-openshift-default-dockercfg-xgptg\""
Nov 01 20:26:07.151030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.150986    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:26:07.178183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178143    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.178388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178283    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctn7\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.178535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178383    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.178535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178450    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.178535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178469    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.179269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178689    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.179269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178710    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.179269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178728    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.179269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.178762    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279485    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279592    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctn7\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279625    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279669    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279694    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279874    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.279918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279901    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279923    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279932    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.279963    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.280086    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.280332    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.280536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.280518    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.281047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.281024    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.282371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.282345    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.282786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.282764    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.340720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.340635    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.340903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.340878    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctn7\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\") pod \"e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") " pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.397143 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.397110    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:07.827518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.827473    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:26:07.828504 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:26:07.828467    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e512e_461c_4c7f_939a_28b2d25b8698.slice/crio-cac84b1935c02d5a2e8ce278a2b9aa37263a64fa33906c306df8f7a91ff345d4 WatchSource:0}: Error finding container cac84b1935c02d5a2e8ce278a2b9aa37263a64fa33906c306df8f7a91ff345d4: Status 404 returned error can't find the container with id cac84b1935c02d5a2e8ce278a2b9aa37263a64fa33906c306df8f7a91ff345d4
Nov 01 20:26:07.830745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.830712    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:26:07.830849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.830787    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:26:07.830849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:07.830829    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:26:08.131026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:08.130946    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerStarted","Data":"a23a318caa4b0aa55cfe88ab20e4819c8986f496f87b520717564631ecd58072"}
Nov 01 20:26:08.131026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:08.130996    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerStarted","Data":"cac84b1935c02d5a2e8ce278a2b9aa37263a64fa33906c306df8f7a91ff345d4"}
Nov 01 20:26:08.248640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:08.248584    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podStartSLOduration=2.248568652 podStartE2EDuration="2.248568652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:08.247799585 +0000 UTC m=+1421.632697177" watchObservedRunningTime="2026-11-01 20:26:08.248568652 +0000 UTC m=+1421.633466243"
Nov 01 20:26:08.398132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:08.398050    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:08.403792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:08.403766    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:09.140038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:09.140010    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:09.141272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:09.141248    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:26:10.862462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.862434    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:26:10.922102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922067    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922111    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922167    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922210    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922240    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922285    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922416    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922462    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922489    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\") pod \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\" (UID: \"1fe1a3a3-f0fc-407c-b43d-3621b7e32706\") "
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922462    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:10.922604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:10.922926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922706    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:10.922926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922833    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:10.922926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.922856    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:10.923084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.923047    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:10.923084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.923070    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:10.923195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.923084    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:10.923195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.923094    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:10.923195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.923103    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:10.924400 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.924377    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:10.924752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.924736    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:26:10.925086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.925066    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:10.925168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:10.925100    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz" pod "1fe1a3a3-f0fc-407c-b43d-3621b7e32706" (UID: "1fe1a3a3-f0fc-407c-b43d-3621b7e32706"). InnerVolumeSpecName "kube-api-access-8pnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:11.024300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.024267    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pnzz\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-kube-api-access-8pnzz\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:11.024300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.024297    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:11.024300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.024308    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:11.024574 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.024318    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe1a3a3-f0fc-407c-b43d-3621b7e32706-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:26:11.154239 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.154150    5066 generic.go:356] "Generic (PLEG): container finished" podID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerID="30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501" exitCode=0
Nov 01 20:26:11.154239 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.154211    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"
Nov 01 20:26:11.154462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.154239    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerDied","Data":"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"}
Nov 01 20:26:11.154462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.154305    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r" event={"ID":"1fe1a3a3-f0fc-407c-b43d-3621b7e32706","Type":"ContainerDied","Data":"af299f8be3824fad5781297aa8ded39debe6356d984e287583ebc3263bb3b921"}
Nov 01 20:26:11.154462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.154340    5066 scope.go:122] "RemoveContainer" containerID="30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"
Nov 01 20:26:11.165300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.165284    5066 scope.go:122] "RemoveContainer" containerID="dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"
Nov 01 20:26:11.174347 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.174327    5066 scope.go:122] "RemoveContainer" containerID="30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"
Nov 01 20:26:11.174622 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:26:11.174599    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501\": container with ID starting with 30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501 not found: ID does not exist" containerID="30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"
Nov 01 20:26:11.174713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.174643    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501"} err="failed to get container status \"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501\": rpc error: code = NotFound desc = could not find container \"30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501\": container with ID starting with 30abed82309439cef43ac19cfc0011a7dd6c431e0be3e77d5362c085e933d501 not found: ID does not exist"
Nov 01 20:26:11.174713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.174667    5066 scope.go:122] "RemoveContainer" containerID="dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"
Nov 01 20:26:11.174920 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:26:11.174898    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb\": container with ID starting with dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb not found: ID does not exist" containerID="dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"
Nov 01 20:26:11.174968 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.174930    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb"} err="failed to get container status \"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb\": rpc error: code = NotFound desc = could not find container \"dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb\": container with ID starting with dbc5d3b162a80298e236e13e482bbe09493b32ee8b83e20ce12b093f73ba5edb not found: ID does not exist"
Nov 01 20:26:11.327435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.327359    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:26:11.330385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:11.330356    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2a4e4430-openshift-default-7dbf8767b7-9vn4r"]
Nov 01 20:26:12.732187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:12.732147    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" path="/var/lib/kubelet/pods/1fe1a3a3-f0fc-407c-b43d-3621b7e32706/volumes"
Nov 01 20:26:13.772132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.772097    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:26:13.772849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.772830    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.772913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.772850    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.772913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.772869    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.772913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.772874    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.773039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.773025    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.778279 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.778255    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.834588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.834557    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:26:13.858766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.858719    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.859163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.859135    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrvk\" (UniqueName: \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.880104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.880071    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-46c731c3\""
Nov 01 20:26:13.880959 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.880937    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-46c731c3-dockercfg-nnj6j\""
Nov 01 20:26:13.957715 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.957673    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:26:13.959094 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.959070    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe1a3a3-f0fc-407c-b43d-3621b7e32706" containerName="istio-proxy"
Nov 01 20:26:13.960634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.960608    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrvk\" (UniqueName: \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.960834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.960812    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.961747 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.961720    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:13.967895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.967871    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:13.991604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:13.991577    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:26:14.017064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.017035    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrvk\" (UniqueName: \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\") pod \"payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:14.062276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.062199    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.062276 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.062246    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmbt\" (UniqueName: \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.095909 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.095876    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:14.168328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.163293    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.168328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.163361    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmbt\" (UniqueName: \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.168328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.164822    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.220249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.220215    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmbt\" (UniqueName: \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\") pod \"payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") " pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.287443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.287409    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:14.481007 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:26:14.480977    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b18623e_9a68_4153_8310_a1ca8bca2c47.slice/crio-ea0584e9b0ee0a0668987b711388c24d2197c1f3e00c1051c141777c555d7214 WatchSource:0}: Error finding container ea0584e9b0ee0a0668987b711388c24d2197c1f3e00c1051c141777c555d7214: Status 404 returned error can't find the container with id ea0584e9b0ee0a0668987b711388c24d2197c1f3e00c1051c141777c555d7214
Nov 01 20:26:14.481769 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.481740    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:26:14.663325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:14.663293    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:26:14.663992 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:26:14.663949    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3421555e_d626_43c8_8b71_caf20af703fa.slice/crio-c4cad3cf2cffe2a97a6d079ed468a212efaf4a989c4bba5162ab43fb1539e3cf WatchSource:0}: Error finding container c4cad3cf2cffe2a97a6d079ed468a212efaf4a989c4bba5162ab43fb1539e3cf: Status 404 returned error can't find the container with id c4cad3cf2cffe2a97a6d079ed468a212efaf4a989c4bba5162ab43fb1539e3cf
Nov 01 20:26:15.195877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.195823    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" event={"ID":"4b18623e-9a68-4153-8310-a1ca8bca2c47","Type":"ContainerStarted","Data":"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"}
Nov 01 20:26:15.196378 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.195926    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" event={"ID":"4b18623e-9a68-4153-8310-a1ca8bca2c47","Type":"ContainerStarted","Data":"ea0584e9b0ee0a0668987b711388c24d2197c1f3e00c1051c141777c555d7214"}
Nov 01 20:26:15.196378 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.196103    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:15.197874 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.197844    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" event={"ID":"3421555e-d626-43c8-8b71-caf20af703fa","Type":"ContainerStarted","Data":"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"}
Nov 01 20:26:15.197998 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.197894    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" event={"ID":"3421555e-d626-43c8-8b71-caf20af703fa","Type":"ContainerStarted","Data":"c4cad3cf2cffe2a97a6d079ed468a212efaf4a989c4bba5162ab43fb1539e3cf"}
Nov 01 20:26:15.197998 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.197924    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:15.310973 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:15.310912    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" podStartSLOduration=2.310898672 podStartE2EDuration="2.310898672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:15.307956443 +0000 UTC m=+1428.692854034" watchObservedRunningTime="2026-11-01 20:26:15.310898672 +0000 UTC m=+1428.695796278"
Nov 01 20:26:21.629244 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:26:21.629209    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:26:26.209674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:26.209641    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:26:26.210385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:26.210363    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:26:26.324867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:26.324816    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" podStartSLOduration=13.324803612 podStartE2EDuration="13.324803612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:15.427142716 +0000 UTC m=+1428.812040304" watchObservedRunningTime="2026-11-01 20:26:26.324803612 +0000 UTC m=+1439.709701203"
Nov 01 20:26:34.803970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.803926    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:26:34.837975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.837941    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902178    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902253    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902316    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902418    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902456    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.903532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.902589    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqp77\" (UniqueName: \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:34.908913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.908887    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-71335882\"/\"openshift-service-ca.crt\""
Nov 01 20:26:34.911903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.909894    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-71335882\"/\"default-dockercfg-26qkb\""
Nov 01 20:26:34.912055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.910044    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-71335882\"/\"e2e-lifecycle-model-71335882-kserve-self-signed-certs\""
Nov 01 20:26:34.914519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.914464    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:26:34.984808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:34.984778    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-71335882\"/\"kube-root-ca.crt\""
Nov 01 20:26:35.014679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.005704    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqp77\" (UniqueName: \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.015395 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.015370    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.015891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.015869    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.016806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.016773    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.017023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.016158    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.017450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.017433    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.018131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.018110    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.018417 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.018048    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.019885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.017731    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.034938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.034879    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.077589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.064956    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.140519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.138406    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqp77\" (UniqueName: \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\") pod \"e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") " pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.324520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.313739    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:35.943751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:35.939284    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:26:36.389592 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:36.389561    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" event={"ID":"15ba9911-cef5-40aa-9351-203bfe11a828","Type":"ContainerStarted","Data":"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"}
Nov 01 20:26:36.389777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:36.389600    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" event={"ID":"15ba9911-cef5-40aa-9351-203bfe11a828","Type":"ContainerStarted","Data":"e2fe814304bc28d6e94cc1db4aa032c5b802500937db2f29f73243c7f25c122b"}
Nov 01 20:26:36.502222 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:36.502165    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" podStartSLOduration=2.502151118 podStartE2EDuration="2.502151118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:36.50166073 +0000 UTC m=+1449.886558322" watchObservedRunningTime="2026-11-01 20:26:36.502151118 +0000 UTC m=+1449.887048708"
Nov 01 20:26:37.315170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:37.315139    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:45.314303 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:45.314265    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:45.327676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:45.327654    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:45.469847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:45.469816    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:26:51.630203 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:26:51.630166    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:26:57.047053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:26:57.047016    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:27:12.067658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.067517    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" containerName="main" containerID="cri-o://d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f" gracePeriod=45
Nov 01 20:27:12.344985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.344957    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:27:12.506231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506191    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506235    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506506 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506302    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\" (UniqueName: \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506506 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506399    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\" (UniqueName: \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506506 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506462    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506479    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:27:12.506688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506523    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\") pod \"37c71721-7dcf-469c-9a22-94c39622d17c\" (UID: \"37c71721-7dcf-469c-9a22-94c39622d17c\") "
Nov 01 20:27:12.506688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506546    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:27:12.506850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.506722    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:27:12.507158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.507128    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.507277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.507187    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.507277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.507203    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.508669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.508604    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:27:12.508772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.508681    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:27:12.508814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.508768    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw" pod "37c71721-7dcf-469c-9a22-94c39622d17c" (UID: "37c71721-7dcf-469c-9a22-94c39622d17c"). InnerVolumeSpecName "kube-api-access-92qfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:27:12.608159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.608084    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c71721-7dcf-469c-9a22-94c39622d17c-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.608159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.608112    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qfw\" (UniqueName: \"kubernetes.io/projected/37c71721-7dcf-469c-9a22-94c39622d17c-kube-api-access-92qfw\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.608159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.608123    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c71721-7dcf-469c-9a22-94c39622d17c-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:12.660908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.660868    5066 generic.go:356] "Generic (PLEG): container finished" podID="37c71721-7dcf-469c-9a22-94c39622d17c" containerID="d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f" exitCode=0
Nov 01 20:27:12.660908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.660911    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"
Nov 01 20:27:12.661163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.660957    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" event={"ID":"37c71721-7dcf-469c-9a22-94c39622d17c","Type":"ContainerDied","Data":"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"}
Nov 01 20:27:12.661163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.661001    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h" event={"ID":"37c71721-7dcf-469c-9a22-94c39622d17c","Type":"ContainerDied","Data":"12640d901eb3f6ab3427b9e79515e5d8da54040fd6ab18c8a1dc060367148d0f"}
Nov 01 20:27:12.661163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.661025    5066 scope.go:122] "RemoveContainer" containerID="d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"
Nov 01 20:27:12.673568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.673368    5066 scope.go:122] "RemoveContainer" containerID="d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"
Nov 01 20:27:12.673762 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:27:12.673737    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f\": container with ID starting with d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f not found: ID does not exist" containerID="d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"
Nov 01 20:27:12.673827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.673777    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f"} err="failed to get container status \"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f\": rpc error: code = NotFound desc = could not find container \"d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f\": container with ID starting with d4d7bbd57412e056602e5129b4ddf34bfd38016da5ed0fe0dddce1e88055816f not found: ID does not exist"
Nov 01 20:27:12.841051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.841016    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:27:12.846288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:12.846251    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/rate-test-model-df41b8-kserve-6cd97cfcf7-f9b9h"]
Nov 01 20:27:14.793048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:14.793009    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" path="/var/lib/kubelet/pods/37c71721-7dcf-469c-9a22-94c39622d17c/volumes"
Nov 01 20:27:21.631091 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:27:21.631051    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:27:23.782688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.782649    5066 generic.go:356] "Generic (PLEG): container finished" podID="783e512e-461c-4c7f-939a-28b2d25b8698" containerID="a23a318caa4b0aa55cfe88ab20e4819c8986f496f87b520717564631ecd58072" exitCode=0
Nov 01 20:27:23.783154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.782692    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerDied","Data":"a23a318caa4b0aa55cfe88ab20e4819c8986f496f87b520717564631ecd58072"}
Nov 01 20:27:23.783241 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.783224    5066 scope.go:122] "RemoveContainer" containerID="a23a318caa4b0aa55cfe88ab20e4819c8986f496f87b520717564631ecd58072"
Nov 01 20:27:23.784882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.784855    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:27:23.784997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.784920    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:27:23.784997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:23.784950    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:27:24.140661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:24.140586    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:27:24.794488 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:24.794454    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerStarted","Data":"837a14b2b4415c1009ae09394a8ee27f6050cf3d0ba2f647c884cc532106c2a8"}
Nov 01 20:27:25.397637 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:25.397603    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:27:25.398781 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:25.398754    5066 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 01 20:27:25.398928 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:25.398815    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 01 20:27:26.398091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:26.398051    5066 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 01 20:27:26.398525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:26.398135    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 01 20:27:26.949507 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:26.949464    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:27:27.014482 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:27.014456    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:27:27.397600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:27.397565    5066 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 01 20:27:27.397769 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:27.397623    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 01 20:27:28.398062 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:28.398026    5066 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 01 20:27:28.398470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:28.398084    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 01 20:27:29.398104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:29.398074    5066 patch_prober.go:29] interesting pod/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 01 20:27:29.398606 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:29.398569    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 01 20:27:30.401950 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:30.401914    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:27:30.402432 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:30.402295    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:27:30.402940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:30.402919    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:27:37.390562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.390489    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:27:37.391523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.391487    5066 cpu_manager.go:414] "containerMap: removing container" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" containerName="main"
Nov 01 20:27:37.391523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.391524    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" containerName="main"
Nov 01 20:27:37.391680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.391669    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c71721-7dcf-469c-9a22-94c39622d17c" containerName="main"
Nov 01 20:27:37.397202 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.397174    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.451243 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.451202    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:27:37.461275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461246    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.461436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461282    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhq9\" (UniqueName: \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.461436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461408    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.461577 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461537    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.461639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461603    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.461709 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.461691    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.494910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.494873    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"kube-root-ca.crt\""
Nov 01 20:27:37.495080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.495061    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"sub-test-model-df41b8-kserve-self-signed-certs\""
Nov 01 20:27:37.498457 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.498431    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"default-dockercfg-rp5nq\""
Nov 01 20:27:37.498457 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.498436    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-df41b8\"/\"openshift-service-ca.crt\""
Nov 01 20:27:37.563603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.563560    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.576216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.576182    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhq9\" (UniqueName: \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.576586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.576559    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.576745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.576729    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.576890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.576874    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.577050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.577036    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.577650 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.577621    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.578374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.578350    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.579362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.579333    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.581650 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.581629    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.582216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.582193    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.634413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.634380    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhq9\" (UniqueName: \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\") pod \"sub-test-model-df41b8-kserve-699d9fd455-h9rg4\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") " pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:37.715287 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:37.715262    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:38.097314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:38.097279    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:27:38.098091 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:27:38.098060    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod167f2990_e934_4d04_8132_83cc7b0b8843.slice/crio-c6ead1eed90305185f35be8df0bb57f06e79aa0c19ff96ba9f3b2479d76c7d44 WatchSource:0}: Error finding container c6ead1eed90305185f35be8df0bb57f06e79aa0c19ff96ba9f3b2479d76c7d44: Status 404 returned error can't find the container with id c6ead1eed90305185f35be8df0bb57f06e79aa0c19ff96ba9f3b2479d76c7d44
Nov 01 20:27:38.942850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:38.942813    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" event={"ID":"167f2990-e934-4d04-8132-83cc7b0b8843","Type":"ContainerStarted","Data":"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"}
Nov 01 20:27:38.943244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:38.942863    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" event={"ID":"167f2990-e934-4d04-8132-83cc7b0b8843","Type":"ContainerStarted","Data":"c6ead1eed90305185f35be8df0bb57f06e79aa0c19ff96ba9f3b2479d76c7d44"}
Nov 01 20:27:39.060402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:39.060345    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" podStartSLOduration=2.060331428 podStartE2EDuration="2.060331428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:39.057748701 +0000 UTC m=+1512.442646292" watchObservedRunningTime="2026-11-01 20:27:39.060331428 +0000 UTC m=+1512.445229019"
Nov 01 20:27:39.715721 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:39.715690    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:46.045713 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.045679    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:27:46.050627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.050603    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.079049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.079016    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq8h\" (UniqueName: \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.079249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.079222    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.105385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.105355    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:27:46.132137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.132108    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-202d23\""
Nov 01 20:27:46.167445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.167410    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-202d23-dockercfg-ft7zg\""
Nov 01 20:27:46.193698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.191603    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.193698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.191846    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq8h\" (UniqueName: \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.193698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.193182    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.212517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.209590    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:27:46.215032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.215006    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.248394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.248366    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq8h\" (UniqueName: \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\") pod \"payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.287910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.287879    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:27:46.293756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.293724    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5ww\" (UniqueName: \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.293917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.293895    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.366742 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.366662    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:46.395899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.395871    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5ww\" (UniqueName: \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.396050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.395998    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.396951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.396924    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.452675 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.452630    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5ww\" (UniqueName: \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\") pod \"payload-processing-e2e-autores-202d23-857bc57847-vct2k\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") " pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.542700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.542667    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:46.747743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.747717    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:27:46.748209 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:27:46.748182    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf734fab_ca0d_4c51_97f9_32d451686b3b.slice/crio-0dc9b5fbd2271a00e06157f0a6883a85bfe3d64d7375d1bb1e3f90ae4c85a7e0 WatchSource:0}: Error finding container 0dc9b5fbd2271a00e06157f0a6883a85bfe3d64d7375d1bb1e3f90ae4c85a7e0: Status 404 returned error can't find the container with id 0dc9b5fbd2271a00e06157f0a6883a85bfe3d64d7375d1bb1e3f90ae4c85a7e0
Nov 01 20:27:46.941470 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:27:46.941437    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2b2186_fe31_4c83_a0aa_60a28a04fd1c.slice/crio-e68de1248a17957f69ff0c35a0a042b9fae3fae0fc49053ab767e1165c82c82a WatchSource:0}: Error finding container e68de1248a17957f69ff0c35a0a042b9fae3fae0fc49053ab767e1165c82c82a: Status 404 returned error can't find the container with id e68de1248a17957f69ff0c35a0a042b9fae3fae0fc49053ab767e1165c82c82a
Nov 01 20:27:46.942919 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:46.942892    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:27:47.012840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.007924    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" event={"ID":"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c","Type":"ContainerStarted","Data":"e68de1248a17957f69ff0c35a0a042b9fae3fae0fc49053ab767e1165c82c82a"}
Nov 01 20:27:47.012840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.009829    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" event={"ID":"cf734fab-ca0d-4c51-97f9-32d451686b3b","Type":"ContainerStarted","Data":"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"}
Nov 01 20:27:47.012840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.009872    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" event={"ID":"cf734fab-ca0d-4c51-97f9-32d451686b3b","Type":"ContainerStarted","Data":"0dc9b5fbd2271a00e06157f0a6883a85bfe3d64d7375d1bb1e3f90ae4c85a7e0"}
Nov 01 20:27:47.012840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.009897    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:47.127542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.127481    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" podStartSLOduration=1.127466635 podStartE2EDuration="1.127466635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:47.124593413 +0000 UTC m=+1520.509491004" watchObservedRunningTime="2026-11-01 20:27:47.127466635 +0000 UTC m=+1520.512364225"
Nov 01 20:27:47.715439 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.715404    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:47.727817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:47.727786    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:48.018336 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:48.018248    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" event={"ID":"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c","Type":"ContainerStarted","Data":"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"}
Nov 01 20:27:48.018529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:48.018405    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:48.029989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:48.029967    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:27:48.130564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:48.130519    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" podStartSLOduration=2.13048864 podStartE2EDuration="2.13048864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:27:48.128987964 +0000 UTC m=+1521.513885561" watchObservedRunningTime="2026-11-01 20:27:48.13048864 +0000 UTC m=+1521.515386231"
Nov 01 20:27:51.630999 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:27:51.630963    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:27:54.531753 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:54.531708    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:27:54.637381 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:54.637348    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:27:58.021638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:58.021609    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:27:59.030031 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.029998    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:27:59.532917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.532869    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" podUID="3421555e-d626-43c8-8b71-caf20af703fa" containerName="payload-processing" containerID="cri-o://60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131" gracePeriod=25
Nov 01 20:27:59.638404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.638337    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" podUID="4b18623e-9a68-4153-8310-a1ca8bca2c47" containerName="payload-pre-processing" containerID="cri-o://c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a" gracePeriod=25
Nov 01 20:27:59.812832 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.812806    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:27:59.910910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.910887    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:27:59.925108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.925086    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\") pod \"3421555e-d626-43c8-8b71-caf20af703fa\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") "
Nov 01 20:27:59.925267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.925254    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\" (UniqueName: \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\") pod \"3421555e-d626-43c8-8b71-caf20af703fa\" (UID: \"3421555e-d626-43c8-8b71-caf20af703fa\") "
Nov 01 20:27:59.925441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.925418    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume" pod "3421555e-d626-43c8-8b71-caf20af703fa" (UID: "3421555e-d626-43c8-8b71-caf20af703fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:27:59.925755 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.925737    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3421555e-d626-43c8-8b71-caf20af703fa-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:27:59.928388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:27:59.928355    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt" pod "3421555e-d626-43c8-8b71-caf20af703fa" (UID: "3421555e-d626-43c8-8b71-caf20af703fa"). InnerVolumeSpecName "kube-api-access-nlmbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:00.027057 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.027013    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\") pod \"4b18623e-9a68-4153-8310-a1ca8bca2c47\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") "
Nov 01 20:28:00.027244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.027161    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\" (UniqueName: \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\") pod \"4b18623e-9a68-4153-8310-a1ca8bca2c47\" (UID: \"4b18623e-9a68-4153-8310-a1ca8bca2c47\") "
Nov 01 20:28:00.027393 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.027369    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume" pod "4b18623e-9a68-4153-8310-a1ca8bca2c47" (UID: "4b18623e-9a68-4153-8310-a1ca8bca2c47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:00.027622 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.027606    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b18623e-9a68-4153-8310-a1ca8bca2c47-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:00.027622 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.027624    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmbt\" (UniqueName: \"kubernetes.io/projected/3421555e-d626-43c8-8b71-caf20af703fa-kube-api-access-nlmbt\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:00.029283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.029253    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk" pod "4b18623e-9a68-4153-8310-a1ca8bca2c47" (UID: "4b18623e-9a68-4153-8310-a1ca8bca2c47"). InnerVolumeSpecName "kube-api-access-dbrvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:00.122808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.122721    5066 generic.go:356] "Generic (PLEG): container finished" podID="4b18623e-9a68-4153-8310-a1ca8bca2c47" containerID="c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a" exitCode=0
Nov 01 20:28:00.122808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.122773    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"
Nov 01 20:28:00.123291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.122814    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" event={"ID":"4b18623e-9a68-4153-8310-a1ca8bca2c47","Type":"ContainerDied","Data":"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"}
Nov 01 20:28:00.123291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.122873    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx" event={"ID":"4b18623e-9a68-4153-8310-a1ca8bca2c47","Type":"ContainerDied","Data":"ea0584e9b0ee0a0668987b711388c24d2197c1f3e00c1051c141777c555d7214"}
Nov 01 20:28:00.123291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.122907    5066 scope.go:122] "RemoveContainer" containerID="c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"
Nov 01 20:28:00.124415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.124389    5066 generic.go:356] "Generic (PLEG): container finished" podID="3421555e-d626-43c8-8b71-caf20af703fa" containerID="60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131" exitCode=0
Nov 01 20:28:00.124415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.124410    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"
Nov 01 20:28:00.124621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.124449    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" event={"ID":"3421555e-d626-43c8-8b71-caf20af703fa","Type":"ContainerDied","Data":"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"}
Nov 01 20:28:00.124621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.124512    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf" event={"ID":"3421555e-d626-43c8-8b71-caf20af703fa","Type":"ContainerDied","Data":"c4cad3cf2cffe2a97a6d079ed468a212efaf4a989c4bba5162ab43fb1539e3cf"}
Nov 01 20:28:00.128541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.128517    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrvk\" (UniqueName: \"kubernetes.io/projected/4b18623e-9a68-4153-8310-a1ca8bca2c47-kube-api-access-dbrvk\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:00.134722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.134701    5066 scope.go:122] "RemoveContainer" containerID="c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"
Nov 01 20:28:00.135008 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:00.134979    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a\": container with ID starting with c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a not found: ID does not exist" containerID="c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"
Nov 01 20:28:00.135061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.135021    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a"} err="failed to get container status \"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a\": rpc error: code = NotFound desc = could not find container \"c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a\": container with ID starting with c2e4bf1de8b8616300b60cd0f385a87e352ff22bf9bd27b645622645ab261f2a not found: ID does not exist"
Nov 01 20:28:00.135061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.135046    5066 scope.go:122] "RemoveContainer" containerID="60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"
Nov 01 20:28:00.144894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.144878    5066 scope.go:122] "RemoveContainer" containerID="60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"
Nov 01 20:28:00.145137 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:00.145117    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131\": container with ID starting with 60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131 not found: ID does not exist" containerID="60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"
Nov 01 20:28:00.145182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.145144    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131"} err="failed to get container status \"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131\": rpc error: code = NotFound desc = could not find container \"60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131\": container with ID starting with 60a6b5b5ba486ab44227ac60b837dfbd92b8a106eac570643dc20f7f75f94131 not found: ID does not exist"
Nov 01 20:28:00.286638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.286605    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:28:00.292253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.292226    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46c731c3-65ffddc6dc-s67nx"]
Nov 01 20:28:00.400228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.400151    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:28:00.403163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.403137    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46c731c3-54fcc7c965-cqxgf"]
Nov 01 20:28:00.733895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.733696    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3421555e-d626-43c8-8b71-caf20af703fa" path="/var/lib/kubelet/pods/3421555e-d626-43c8-8b71-caf20af703fa/volumes"
Nov 01 20:28:00.734308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:00.734285    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b18623e-9a68-4153-8310-a1ca8bca2c47" path="/var/lib/kubelet/pods/4b18623e-9a68-4153-8310-a1ca8bca2c47/volumes"
Nov 01 20:28:13.369818 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.369784    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:28:13.370476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370456    5066 cpu_manager.go:414] "containerMap: removing container" podUID="3421555e-d626-43c8-8b71-caf20af703fa" containerName="payload-processing"
Nov 01 20:28:13.370557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370477    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3421555e-d626-43c8-8b71-caf20af703fa" containerName="payload-processing"
Nov 01 20:28:13.370557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370529    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4b18623e-9a68-4153-8310-a1ca8bca2c47" containerName="payload-pre-processing"
Nov 01 20:28:13.370557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370536    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b18623e-9a68-4153-8310-a1ca8bca2c47" containerName="payload-pre-processing"
Nov 01 20:28:13.370685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370649    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b18623e-9a68-4153-8310-a1ca8bca2c47" containerName="payload-pre-processing"
Nov 01 20:28:13.370685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.370664    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="3421555e-d626-43c8-8b71-caf20af703fa" containerName="payload-processing"
Nov 01 20:28:13.375697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.375678    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.427892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.427858    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-4bc05b0bda98\""
Nov 01 20:28:13.427892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.427896    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-cd18c1528ab8\""
Nov 01 20:28:13.428124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.427899    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-e385637602d5\""
Nov 01 20:28:13.428124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.427927    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\""
Nov 01 20:28:13.430229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.430200    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:28:13.451113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.451085    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-292313e3f8da\""
Nov 01 20:28:13.451331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.451093    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\""
Nov 01 20:28:13.451331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.451098    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\""
Nov 01 20:28:13.474715 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.474691    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\""
Nov 01 20:28:13.474840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.474719    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-t8dl8\""
Nov 01 20:28:13.497638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.497598    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8vw\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.497791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.497644    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.497791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.497769    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.497950 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.497928    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.599013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.598974    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.599182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.599157    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.599284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.599257    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8vw\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.599338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.599303    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.600010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.599981    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.600125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.600061    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.602172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.602149    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.656246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.656156    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8vw\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:13.690849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:13.690820    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:14.071667 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:14.071636    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:28:14.072032 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:14.072010    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc8d863_3b27_472e_b9a2_93545c8cea94.slice/crio-49ec3ca0426f6db4c1dc74954e38993ec78a9b9e343a86336717dc987cd017a4 WatchSource:0}: Error finding container 49ec3ca0426f6db4c1dc74954e38993ec78a9b9e343a86336717dc987cd017a4: Status 404 returned error can't find the container with id 49ec3ca0426f6db4c1dc74954e38993ec78a9b9e343a86336717dc987cd017a4
Nov 01 20:28:14.226627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:14.226590    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" event={"ID":"6fc8d863-3b27-472e-b9a2-93545c8cea94","Type":"ContainerStarted","Data":"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"}
Nov 01 20:28:14.226801 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:14.226654    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" event={"ID":"6fc8d863-3b27-472e-b9a2-93545c8cea94","Type":"ContainerStarted","Data":"49ec3ca0426f6db4c1dc74954e38993ec78a9b9e343a86336717dc987cd017a4"}
Nov 01 20:28:14.226801 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:14.226679    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:14.340459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:14.340365    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" podStartSLOduration=1.340351943 podStartE2EDuration="1.340351943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:14.338340181 +0000 UTC m=+1547.723237787" watchObservedRunningTime="2026-11-01 20:28:14.340351943 +0000 UTC m=+1547.725249576"
Nov 01 20:28:16.536431 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.536393    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:16.545015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.544979    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.597221 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.597190    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:16.633706 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.633674    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzc4j\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.633864 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.633834    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.633906 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.633888    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.634005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.633982    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.641579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.641543    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a3c632f04fca\""
Nov 01 20:28:16.641579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.641564    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 20:28:16.641579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.641552    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 20:28:16.641785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.641554    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 20:28:16.641785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.641550    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 20:28:16.735282 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.735250    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzc4j\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.735466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.735338    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.735466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.735370    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.735466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.735397    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.736056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.736029    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.736199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.736179    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.737876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.737853    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.830343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.830257    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzc4j\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:16.864228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:16.864193    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:17.274913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:17.274881    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:17.275806 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:17.275779    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542ef104_a3c0_48b3_8eb9_ab73f9de30a7.slice/crio-53759ee43341ca94993d353ca76a04f1b471f3732d4c10d3cc2cecdefa06e381 WatchSource:0}: Error finding container 53759ee43341ca94993d353ca76a04f1b471f3732d4c10d3cc2cecdefa06e381: Status 404 returned error can't find the container with id 53759ee43341ca94993d353ca76a04f1b471f3732d4c10d3cc2cecdefa06e381
Nov 01 20:28:17.296388 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:17.296349    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:28:17.407453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:17.407389    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:28:18.261097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:18.261045    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" event={"ID":"542ef104-a3c0-48b3-8eb9-ab73f9de30a7","Type":"ContainerStarted","Data":"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"}
Nov 01 20:28:18.261599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:18.261117    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" event={"ID":"542ef104-a3c0-48b3-8eb9-ab73f9de30a7","Type":"ContainerStarted","Data":"53759ee43341ca94993d353ca76a04f1b471f3732d4c10d3cc2cecdefa06e381"}
Nov 01 20:28:18.261599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:18.261166    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:18.375776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:18.375725    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" podStartSLOduration=2.375711013 podStartE2EDuration="2.375711013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:18.372754773 +0000 UTC m=+1551.757652377" watchObservedRunningTime="2026-11-01 20:28:18.375711013 +0000 UTC m=+1551.760608600"
Nov 01 20:28:18.438149 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.438110    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-a3c632f04fca: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:18.438149 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.438138    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:18.438376 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.438197    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials podName:542ef104-a3c0-48b3-8eb9-ab73f9de30a7 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:18.93817972 +0000 UTC m=+1552.323077289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7") : secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:18.955405 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.955369    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-a3c632f04fca: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:18.955405 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.955399    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:18.955707 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:18.955449    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials podName:542ef104-a3c0-48b3-8eb9-ab73f9de30a7 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:19.955434684 +0000 UTC m=+1553.340332257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7") : secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:19.968314 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:19.968277    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-a3c632f04fca: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:19.968314 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:19.968307    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:19.968859 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:19.968373    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials podName:542ef104-a3c0-48b3-8eb9-ab73f9de30a7 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:21.968355387 +0000 UTC m=+1555.353252975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7") : secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:20.064155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:20.064117    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:28:20.064375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:20.064350    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy" containerID="cri-o://837a14b2b4415c1009ae09394a8ee27f6050cf3d0ba2f647c884cc532106c2a8" gracePeriod=30
Nov 01 20:28:21.632038 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:21.631999    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:28:22.005120 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:22.005085    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-a3c632f04fca: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:22.005120 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:22.005114    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:22.005322 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:22.005174    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials podName:542ef104-a3c0-48b3-8eb9-ab73f9de30a7 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:26.005155475 +0000 UTC m=+1559.390053064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7") : secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:22.297097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:22.297004    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing" containerID="cri-o://81dac954220e15a1a327db8fdbb0bf1cda5d468c48f1096eb704cabb70f49d00" gracePeriod=25
Nov 01 20:28:22.407793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:22.407752    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing" containerID="cri-o://2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2" gracePeriod=25
Nov 01 20:28:25.237452 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.237423    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:28:25.319679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.319643    5066 generic.go:356] "Generic (PLEG): container finished" podID="783e512e-461c-4c7f-939a-28b2d25b8698" containerID="837a14b2b4415c1009ae09394a8ee27f6050cf3d0ba2f647c884cc532106c2a8" exitCode=0
Nov 01 20:28:25.319908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.319840    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerDied","Data":"837a14b2b4415c1009ae09394a8ee27f6050cf3d0ba2f647c884cc532106c2a8"}
Nov 01 20:28:25.319908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.319902    5066 scope.go:122] "RemoveContainer" containerID="a23a318caa4b0aa55cfe88ab20e4819c8986f496f87b520717564631ecd58072"
Nov 01 20:28:25.348438 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.348059    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.348438 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.348097    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.348438 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.348179    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:25.848159488 +0000 UTC m=+1559.233057071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.355946 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.355917    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:28:25.540997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.540964    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:25.541705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.541672    5066 cpu_manager.go:414] "containerMap: removing container" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:25.541705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.541704    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:25.541913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.541900    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:25.545997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.545974    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.549544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549518    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549570    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549702 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549650    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549764    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549860 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549804    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549888    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.549940 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.549898    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:25.550069 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550008    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.550069 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550038    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.550069 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550061    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\") pod \"783e512e-461c-4c7f-939a-28b2d25b8698\" (UID: \"783e512e-461c-4c7f-939a-28b2d25b8698\") "
Nov 01 20:28:25.550214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550067    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:25.550367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550306    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:25.550577 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550538    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:25.550777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550758    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.550840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550783    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.550840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550798    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.550840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.550814    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e512e-461c-4c7f-939a-28b2d25b8698-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.551589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.551445    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:25.552651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.552626    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:28:25.552912 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.552895    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "kube-api-access-9ctn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:25.553071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.553058    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:25.554163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.554139    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy" pod "783e512e-461c-4c7f-939a-28b2d25b8698" (UID: "783e512e-461c-4c7f-939a-28b2d25b8698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:25.601963 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.601899    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:25.651797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.651764    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vzl\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.651960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.651801    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.652042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.651962    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.652153 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652137    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.652216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652192    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctn7\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-kube-api-access-9ctn7\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.652263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652222    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.652263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652250    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e512e-461c-4c7f-939a-28b2d25b8698-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.652363 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652266    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e512e-461c-4c7f-939a-28b2d25b8698-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.652363 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.652280    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e512e-461c-4c7f-939a-28b2d25b8698-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:25.753775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.753741    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.753922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.753890    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.753922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.753919    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vzl\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.754016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.753937    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.754670 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.754643    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.754915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.754891    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.756397 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.756375    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.811640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.811573    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vzl\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:25.855188 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.855119    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.855188 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.855154    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.855361 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:25.855209    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:26.855194677 +0000 UTC m=+1560.240092260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:25.884853 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:25.884827    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:26.058616 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.058588    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-a3c632f04fca: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:26.058616 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.058611    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr: secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:26.058790 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.058670    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials podName:542ef104-a3c0-48b3-8eb9-ab73f9de30a7 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:34.058655698 +0000 UTC m=+1567.443553288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7") : secret "agc-cred-a3c632f04fca" not found
Nov 01 20:28:26.261634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.261609    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:26.264415 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:26.264387    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014a99d7_70cd_4e7c_ba07_9e9b83204bad.slice/crio-d576c5fede7fec408e8781050bd8865c4b1d41319ce7df3c4b4ab87124f2244e WatchSource:0}: Error finding container d576c5fede7fec408e8781050bd8865c4b1d41319ce7df3c4b4ab87124f2244e: Status 404 returned error can't find the container with id d576c5fede7fec408e8781050bd8865c4b1d41319ce7df3c4b4ab87124f2244e
Nov 01 20:28:26.329383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.329345    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" event={"ID":"014a99d7-70cd-4e7c-ba07-9e9b83204bad","Type":"ContainerStarted","Data":"d576c5fede7fec408e8781050bd8865c4b1d41319ce7df3c4b4ab87124f2244e"}
Nov 01 20:28:26.331038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.331013    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"
Nov 01 20:28:26.331138 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.331042    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q" event={"ID":"783e512e-461c-4c7f-939a-28b2d25b8698","Type":"ContainerDied","Data":"cac84b1935c02d5a2e8ce278a2b9aa37263a64fa33906c306df8f7a91ff345d4"}
Nov 01 20:28:26.331138 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.331098    5066 scope.go:122] "RemoveContainer" containerID="837a14b2b4415c1009ae09394a8ee27f6050cf3d0ba2f647c884cc532106c2a8"
Nov 01 20:28:26.559176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.559094    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:28:26.565936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.565906    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-46c731c3-gw-openshift-default-6d64986f49-xdz5q"]
Nov 01 20:28:26.741347 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:26.741306    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" path="/var/lib/kubelet/pods/783e512e-461c-4c7f-939a-28b2d25b8698/volumes"
Nov 01 20:28:26.867433 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.867355    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:26.867433 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.867388    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:26.867644 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:26.867453    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:28.867437733 +0000 UTC m=+1562.252335303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:27.013883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:27.013851    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:27.016559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:27.016534    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:27.340100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:27.340068    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" event={"ID":"014a99d7-70cd-4e7c-ba07-9e9b83204bad","Type":"ContainerStarted","Data":"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"}
Nov 01 20:28:27.340482 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:27.340164    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:27.454141 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:27.454092    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" podStartSLOduration=2.454078267 podStartE2EDuration="2.454078267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:27.453028976 +0000 UTC m=+1560.837926575" watchObservedRunningTime="2026-11-01 20:28:27.454078267 +0000 UTC m=+1560.838975926"
Nov 01 20:28:28.894667 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:28.894629    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:28.894667 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:28.894664    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:28.895194 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:28.894726    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:32.894709276 +0000 UTC m=+1566.279606863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:31.553212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:31.553167    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:31.553691 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:31.553225    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:32.015027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.014985    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" podUID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" containerName="payload-processing" containerID="cri-o://74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94" gracePeriod=25
Nov 01 20:28:32.289463 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.289439    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:32.338641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.338613    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\") pod \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") "
Nov 01 20:28:32.338770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.338699    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\") pod \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") "
Nov 01 20:28:32.338770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.338731    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\") pod \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") "
Nov 01 20:28:32.338858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.338816    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\") pod \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\" (UID: \"542ef104-a3c0-48b3-8eb9-ab73f9de30a7\") "
Nov 01 20:28:32.339061 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.339037    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume" pod "542ef104-a3c0-48b3-8eb9-ab73f9de30a7" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:32.339288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.339266    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:32.339288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.339265    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume" pod "542ef104-a3c0-48b3-8eb9-ab73f9de30a7" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:32.340990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.340969    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials" pod "542ef104-a3c0-48b3-8eb9-ab73f9de30a7" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:32.341120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.341103    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j" pod "542ef104-a3c0-48b3-8eb9-ab73f9de30a7" (UID: "542ef104-a3c0-48b3-8eb9-ab73f9de30a7"). InnerVolumeSpecName "kube-api-access-qzc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:32.384886 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.384856    5066 generic.go:356] "Generic (PLEG): container finished" podID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" containerID="74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94" exitCode=0
Nov 01 20:28:32.384979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.384904    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"
Nov 01 20:28:32.384979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.384940    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" event={"ID":"542ef104-a3c0-48b3-8eb9-ab73f9de30a7","Type":"ContainerDied","Data":"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"}
Nov 01 20:28:32.385054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.384995    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr" event={"ID":"542ef104-a3c0-48b3-8eb9-ab73f9de30a7","Type":"ContainerDied","Data":"53759ee43341ca94993d353ca76a04f1b471f3732d4c10d3cc2cecdefa06e381"}
Nov 01 20:28:32.385054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.385024    5066 scope.go:122] "RemoveContainer" containerID="74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"
Nov 01 20:28:32.395760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.395743    5066 scope.go:122] "RemoveContainer" containerID="74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"
Nov 01 20:28:32.396032 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:32.396015    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94\": container with ID starting with 74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94 not found: ID does not exist" containerID="74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"
Nov 01 20:28:32.396100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.396038    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94"} err="failed to get container status \"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94\": rpc error: code = NotFound desc = could not find container \"74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94\": container with ID starting with 74209d8f0fbf6df3e8135364e835da575a0cc6bf26c996050a5ec92199214f94 not found: ID does not exist"
Nov 01 20:28:32.440862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.440828    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:32.440862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.440865    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:32.441058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.440880    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzc4j\" (UniqueName: \"kubernetes.io/projected/542ef104-a3c0-48b3-8eb9-ab73f9de30a7-kube-api-access-qzc4j\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:32.514257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.514222    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:32.518882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.518851    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7d56xkfwr"]
Nov 01 20:28:32.732865 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:32.732827    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" path="/var/lib/kubelet/pods/542ef104-a3c0-48b3-8eb9-ab73f9de30a7/volumes"
Nov 01 20:28:32.947377 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:32.947343    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:32.947377 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:32.947383    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:32.947599 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:32.947452    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:40.947431679 +0000 UTC m=+1574.332329266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:33.646541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.646503    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:28:33.647176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647162    5066 cpu_manager.go:414] "containerMap: removing container" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:33.647223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647178    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:33.647223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647189    5066 cpu_manager.go:414] "containerMap: removing container" podUID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" containerName="payload-processing"
Nov 01 20:28:33.647223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647194    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" containerName="payload-processing"
Nov 01 20:28:33.647319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647310    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="542ef104-a3c0-48b3-8eb9-ab73f9de30a7" containerName="payload-processing"
Nov 01 20:28:33.647355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.647324    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e512e-461c-4c7f-939a-28b2d25b8698" containerName="istio-proxy"
Nov 01 20:28:33.652261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.652244    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.707292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.707258    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:28:33.752478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.752447    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7387069c-dockercfg-gc8w6\""
Nov 01 20:28:33.752883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.752515    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7387069c\""
Nov 01 20:28:33.764179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.764143    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmfs\" (UniqueName: \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.764537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.764482    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.830997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.830946    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:28:33.841232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.840762    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:33.866591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.866545    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmfs\" (UniqueName: \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.867563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.867532    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.872655 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.872627    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.876251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.876222    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:28:33.925304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.925232    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmfs\" (UniqueName: \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\") pod \"payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.967753 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.967721    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:33.970260 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.970227    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjnj\" (UniqueName: \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:33.970389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:33.970282    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.071423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.071375    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjnj\" (UniqueName: \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.071623 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.071443    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.072268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.072242    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.131267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.131226    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjnj\" (UniqueName: \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\") pod \"payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") " pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.191737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.191657    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:34.350683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.350639    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:28:34.352242 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:34.352216    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b13025e_eeb6_4d80_815d_765ae432cb88.slice/crio-727b827a5dbb94e79dd3d056b4b341de08c5b22ba73f5425dcaa6ccaaeee6508 WatchSource:0}: Error finding container 727b827a5dbb94e79dd3d056b4b341de08c5b22ba73f5425dcaa6ccaaeee6508: Status 404 returned error can't find the container with id 727b827a5dbb94e79dd3d056b4b341de08c5b22ba73f5425dcaa6ccaaeee6508
Nov 01 20:28:34.401792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.401767    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" event={"ID":"0b13025e-eeb6-4d80-815d-765ae432cb88","Type":"ContainerStarted","Data":"727b827a5dbb94e79dd3d056b4b341de08c5b22ba73f5425dcaa6ccaaeee6508"}
Nov 01 20:28:34.575809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:34.575783    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:28:34.576286 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:34.576257    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653918fe_ba45_4bbe_b860_60b3fb361de9.slice/crio-520ae0e553c3ff09241e2a8ee9ba0c5169b892fcdda98e4c3271b1352a8d0ad5 WatchSource:0}: Error finding container 520ae0e553c3ff09241e2a8ee9ba0c5169b892fcdda98e4c3271b1352a8d0ad5: Status 404 returned error can't find the container with id 520ae0e553c3ff09241e2a8ee9ba0c5169b892fcdda98e4c3271b1352a8d0ad5
Nov 01 20:28:35.411226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.411193    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" event={"ID":"0b13025e-eeb6-4d80-815d-765ae432cb88","Type":"ContainerStarted","Data":"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"}
Nov 01 20:28:35.411937 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.411256    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:35.413078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.413044    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" event={"ID":"653918fe-ba45-4bbe-b860-60b3fb361de9","Type":"ContainerStarted","Data":"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"}
Nov 01 20:28:35.413204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.413097    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" event={"ID":"653918fe-ba45-4bbe-b860-60b3fb361de9","Type":"ContainerStarted","Data":"520ae0e553c3ff09241e2a8ee9ba0c5169b892fcdda98e4c3271b1352a8d0ad5"}
Nov 01 20:28:35.413204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.413117    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:35.524178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.524114    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" podStartSLOduration=2.524096107 podStartE2EDuration="2.524096107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:35.522284769 +0000 UTC m=+1568.907182354" watchObservedRunningTime="2026-11-01 20:28:35.524096107 +0000 UTC m=+1568.908993702"
Nov 01 20:28:35.637168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:35.637105    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" podStartSLOduration=2.637085429 podStartE2EDuration="2.637085429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:35.633838422 +0000 UTC m=+1569.018736016" watchObservedRunningTime="2026-11-01 20:28:35.637085429 +0000 UTC m=+1569.021983020"
Nov 01 20:28:38.350072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:38.349979    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:40.977051 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:40.977018    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:40.977051 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:40.977052    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:40.977477 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:40.977104    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:56.977090151 +0000 UTC m=+1590.361987721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:41.552932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:41.552893    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:41.552932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:41.552923    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:42.467685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.467648    5066 generic.go:356] "Generic (PLEG): container finished" podID="524a98e1-69f6-4871-b339-49cfc30612e3" containerID="81dac954220e15a1a327db8fdbb0bf1cda5d468c48f1096eb704cabb70f49d00" exitCode=0
Nov 01 20:28:42.468109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.467703    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" event={"ID":"524a98e1-69f6-4871-b339-49cfc30612e3","Type":"ContainerDied","Data":"81dac954220e15a1a327db8fdbb0bf1cda5d468c48f1096eb704cabb70f49d00"}
Nov 01 20:28:42.547152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.547117    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:42.548726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.548694    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:28:42.583119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.583096    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:28:42.624647 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.624609    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:28:42.626577 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.626547    5066 cpu_manager.go:414] "containerMap: removing container" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing"
Nov 01 20:28:42.626736 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.626722    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing"
Nov 01 20:28:42.627122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.627100    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" containerName="payload-processing"
Nov 01 20:28:42.633321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.633282    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.657078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.657051    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:28:42.685361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.685333    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-caf6c2f34a9e\""
Nov 01 20:28:42.702791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.702766    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\") pod \"524a98e1-69f6-4871-b339-49cfc30612e3\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") "
Nov 01 20:28:42.703000 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.702980    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\" (UniqueName: \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\") pod \"524a98e1-69f6-4871-b339-49cfc30612e3\" (UID: \"524a98e1-69f6-4871-b339-49cfc30612e3\") "
Nov 01 20:28:42.703191 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.703159    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume" pod "524a98e1-69f6-4871-b339-49cfc30612e3" (UID: "524a98e1-69f6-4871-b339-49cfc30612e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:42.703793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.703771    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/524a98e1-69f6-4871-b339-49cfc30612e3-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:42.705912 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.705883    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg" pod "524a98e1-69f6-4871-b339-49cfc30612e3" (UID: "524a98e1-69f6-4871-b339-49cfc30612e3"). InnerVolumeSpecName "kube-api-access-lm8xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:42.708567 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.708543    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:28:42.804766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.804685    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\") pod \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") "
Nov 01 20:28:42.804922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.804886    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\" (UniqueName: \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\") pod \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\" (UID: \"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d\") "
Nov 01 20:28:42.805063 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805040    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume" pod "9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" (UID: "9fc2ebd1-e95e-47af-bd4e-02fe0132a39d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:42.805140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805120    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474qp\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.805277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805261    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.805448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805426    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.805560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805543    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.805629 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805618    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8xg\" (UniqueName: \"kubernetes.io/projected/524a98e1-69f6-4871-b339-49cfc30612e3-kube-api-access-lm8xg\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:42.805665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.805631    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:42.807009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.806987    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn" pod "9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" (UID: "9fc2ebd1-e95e-47af-bd4e-02fe0132a39d"). InnerVolumeSpecName "kube-api-access-t77kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:42.906639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.906604    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.906839 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.906777    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.906910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.906852    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.906910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.906896    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474qp\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.907073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.907059    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77kn\" (UniqueName: \"kubernetes.io/projected/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d-kube-api-access-t77kn\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:42.907817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.907789    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.907928 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.907883    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.910483 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.910452    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:42.963958 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:42.963919    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-474qp\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:43.253294 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.253263    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:43.475743 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.475709    5066 generic.go:356] "Generic (PLEG): container finished" podID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerID="2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2" exitCode=0
Nov 01 20:28:43.476212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.475765    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"
Nov 01 20:28:43.476212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.475799    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" event={"ID":"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d","Type":"ContainerDied","Data":"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"}
Nov 01 20:28:43.476212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.475864    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj" event={"ID":"9fc2ebd1-e95e-47af-bd4e-02fe0132a39d","Type":"ContainerDied","Data":"5c130102c2fe1cc8598fcc4c49176aaef55fb22151f291d333ccc6b2674ce5c7"}
Nov 01 20:28:43.476212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.475891    5066 scope.go:122] "RemoveContainer" containerID="2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"
Nov 01 20:28:43.477773 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.477751    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp" event={"ID":"524a98e1-69f6-4871-b339-49cfc30612e3","Type":"ContainerDied","Data":"406af13af003f5668939d630451d58494846eaa8217e7a82c0d9979490708339"}
Nov 01 20:28:43.477873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.477844    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"
Nov 01 20:28:43.487809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.487790    5066 scope.go:122] "RemoveContainer" containerID="2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"
Nov 01 20:28:43.488115 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:43.488087    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2\": container with ID starting with 2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2 not found: ID does not exist" containerID="2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"
Nov 01 20:28:43.488187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.488126    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2"} err="failed to get container status \"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2\": rpc error: code = NotFound desc = could not find container \"2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2\": container with ID starting with 2938bb77b986e26eef841f700dfcda7dee11e0590ea72a0d8198f180cd896bc2 not found: ID does not exist"
Nov 01 20:28:43.488187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.488149    5066 scope.go:122] "RemoveContainer" containerID="81dac954220e15a1a327db8fdbb0bf1cda5d468c48f1096eb704cabb70f49d00"
Nov 01 20:28:43.631602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.631573    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:28:43.632136 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:43.632110    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8b9863_69b2_4ff6_a302_05b1e094d22f.slice/crio-1ab7894da2cd2478c0f6dad028580b08ddea5ada0054d8fbebd9cdaa36db628f WatchSource:0}: Error finding container 1ab7894da2cd2478c0f6dad028580b08ddea5ada0054d8fbebd9cdaa36db628f: Status 404 returned error can't find the container with id 1ab7894da2cd2478c0f6dad028580b08ddea5ada0054d8fbebd9cdaa36db628f
Nov 01 20:28:43.641247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.641225    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:28:43.653438 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.653414    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71335882-7c67fb47b9-dlvjp"]
Nov 01 20:28:43.750793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.750764    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:28:43.753829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:43.753806    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71335882-5bb678b4f5-cqqhj"]
Nov 01 20:28:44.487184 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.487143    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" event={"ID":"7f8b9863-69b2-4ff6-a302-05b1e094d22f","Type":"ContainerStarted","Data":"a01be44721946c78057450816ae2a2191ebc573b0b9236a3dd2569a1ffe07194"}
Nov 01 20:28:44.487677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.487212    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:44.487677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.487239    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" event={"ID":"7f8b9863-69b2-4ff6-a302-05b1e094d22f","Type":"ContainerStarted","Data":"1ab7894da2cd2478c0f6dad028580b08ddea5ada0054d8fbebd9cdaa36db628f"}
Nov 01 20:28:44.603556 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.603472    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" podStartSLOduration=2.6034574729999997 podStartE2EDuration="2.603457473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:44.599700895 +0000 UTC m=+1577.984598496" watchObservedRunningTime="2026-11-01 20:28:44.603457473 +0000 UTC m=+1577.988355063"
Nov 01 20:28:44.732239 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.732206    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524a98e1-69f6-4871-b339-49cfc30612e3" path="/var/lib/kubelet/pods/524a98e1-69f6-4871-b339-49cfc30612e3/volumes"
Nov 01 20:28:44.732697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:44.732677    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" path="/var/lib/kubelet/pods/9fc2ebd1-e95e-47af-bd4e-02fe0132a39d/volumes"
Nov 01 20:28:46.423011 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:46.422977    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:28:46.423531 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:46.423043    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:28:47.547884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.547812    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing" containerID="cri-o://ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee" gracePeriod=25
Nov 01 20:28:47.818994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.818972    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:47.867155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867127    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\") pod \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") "
Nov 01 20:28:47.867307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867162    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\") pod \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") "
Nov 01 20:28:47.867307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867216    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\") pod \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") "
Nov 01 20:28:47.867307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867298    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\") pod \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\" (UID: \"014a99d7-70cd-4e7c-ba07-9e9b83204bad\") "
Nov 01 20:28:47.867641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867610    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume" pod "014a99d7-70cd-4e7c-ba07-9e9b83204bad" (UID: "014a99d7-70cd-4e7c-ba07-9e9b83204bad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:47.867641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867621    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume" pod "014a99d7-70cd-4e7c-ba07-9e9b83204bad" (UID: "014a99d7-70cd-4e7c-ba07-9e9b83204bad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:47.867826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867808    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:47.867882 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.867828    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/014a99d7-70cd-4e7c-ba07-9e9b83204bad-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:47.869576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.869550    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials" pod "014a99d7-70cd-4e7c-ba07-9e9b83204bad" (UID: "014a99d7-70cd-4e7c-ba07-9e9b83204bad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:47.869733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.869714    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl" pod "014a99d7-70cd-4e7c-ba07-9e9b83204bad" (UID: "014a99d7-70cd-4e7c-ba07-9e9b83204bad"). InnerVolumeSpecName "kube-api-access-c8vzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:47.969404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.969370    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vzl\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-kube-api-access-c8vzl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:47.969404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:47.969401    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/014a99d7-70cd-4e7c-ba07-9e9b83204bad-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:28:48.519412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.519379    5066 generic.go:356] "Generic (PLEG): container finished" podID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerID="ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee" exitCode=0
Nov 01 20:28:48.519611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.519421    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"
Nov 01 20:28:48.519611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.519465    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" event={"ID":"014a99d7-70cd-4e7c-ba07-9e9b83204bad","Type":"ContainerDied","Data":"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"}
Nov 01 20:28:48.519611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.519532    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w" event={"ID":"014a99d7-70cd-4e7c-ba07-9e9b83204bad","Type":"ContainerDied","Data":"d576c5fede7fec408e8781050bd8865c4b1d41319ce7df3c4b4ab87124f2244e"}
Nov 01 20:28:48.519611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.519557    5066 scope.go:122] "RemoveContainer" containerID="ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"
Nov 01 20:28:48.530443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.530424    5066 scope.go:122] "RemoveContainer" containerID="ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"
Nov 01 20:28:48.530741 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:48.530721    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee\": container with ID starting with ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee not found: ID does not exist" containerID="ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"
Nov 01 20:28:48.530810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.530750    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee"} err="failed to get container status \"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee\": rpc error: code = NotFound desc = could not find container \"ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee\": container with ID starting with ed699e3b6e8ce7ffa1079e72db5ee310873097543fd2d8f88da62dfa522907ee not found: ID does not exist"
Nov 01 20:28:48.682789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.682755    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:48.732321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:48.732281    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dc9k2w"]
Nov 01 20:28:49.471312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.471280    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:28:49.471907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.471892    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing"
Nov 01 20:28:49.471958 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.471908    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing"
Nov 01 20:28:49.471958 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.471920    5066 cpu_manager.go:414] "containerMap: removing container" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing"
Nov 01 20:28:49.471958 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.471927    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing"
Nov 01 20:28:49.472063 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.472048    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc2ebd1-e95e-47af-bd4e-02fe0132a39d" containerName="payload-pre-processing"
Nov 01 20:28:49.472063 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.472057    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" containerName="payload-processing"
Nov 01 20:28:49.476771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.476753    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.528674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.528644    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 20:28:49.528829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.528647    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 20:28:49.528829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.528787    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 20:28:49.528829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.528805    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q7njp\""
Nov 01 20:28:49.583898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.583857    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:28:49.588412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588380    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588470    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588517    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588609    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588643    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588722    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbc55\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.588894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.588874    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.690670 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.690628    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.690682    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.690767    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc55\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.690806    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.690928    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.691003    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.691032    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.691671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.691646    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.693267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.693243    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.693689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.693665    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.701568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.694201    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.701568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.694239    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.747368 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.747290    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.747537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.747486    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc55\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p6kz\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:49.793675 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:49.793645    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:50.274866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:50.274703    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:28:50.276832 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:28:50.276802    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc9a9a4_0a0f_4499_9701_9129ad9b573d.slice/crio-458482adaa5a5a18f9345c7a8e312c8200fffb134d27a2ae6a56a9cd1f580134 WatchSource:0}: Error finding container 458482adaa5a5a18f9345c7a8e312c8200fffb134d27a2ae6a56a9cd1f580134: Status 404 returned error can't find the container with id 458482adaa5a5a18f9345c7a8e312c8200fffb134d27a2ae6a56a9cd1f580134
Nov 01 20:28:50.279135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:50.279113    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:28:50.534791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:50.534707    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" event={"ID":"ccc9a9a4-0a0f-4499-9701-9129ad9b573d","Type":"ContainerStarted","Data":"458482adaa5a5a18f9345c7a8e312c8200fffb134d27a2ae6a56a9cd1f580134"}
Nov 01 20:28:50.731880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:50.731840    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014a99d7-70cd-4e7c-ba07-9e9b83204bad" path="/var/lib/kubelet/pods/014a99d7-70cd-4e7c-ba07-9e9b83204bad/volumes"
Nov 01 20:28:51.629509 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:51.629454    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:28:54.771325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:54.771290    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:28:54.771652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:54.771359    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:28:55.499776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.499747    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:28:55.576608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.576570    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" event={"ID":"ccc9a9a4-0a0f-4499-9701-9129ad9b573d","Type":"ContainerStarted","Data":"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"}
Nov 01 20:28:55.576775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.576630    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:55.578738 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.578714    5066 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9p6kz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:28:55.578854 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.578758    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:28:55.670097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.670063    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:28:55.731100 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:55.731042    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" podStartSLOduration=2.240189958 podStartE2EDuration="6.731026743s" totalImagesPullingTime="4.490836785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:55.727542237 +0000 UTC m=+1589.112439822" watchObservedRunningTime="2026-11-01 20:28:55.731026743 +0000 UTC m=+1589.115924340"
Nov 01 20:28:56.583613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:28:56.583583    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:28:57.004800 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:57.004769    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-292313e3f8da: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:57.004800 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:57.004802    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c: secret "agc-cred-292313e3f8da" not found
Nov 01 20:28:57.004991 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:28:57.004853    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials podName:6fc8d863-3b27-472e-b9a2-93545c8cea94 nodeName:}" failed. No retries permitted until 2026-11-01 20:29:29.004839705 +0000 UTC m=+1622.389737274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94") : secret "agc-cred-292313e3f8da" not found
Nov 01 20:29:00.670377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:00.670323    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" podUID="6fc8d863-3b27-472e-b9a2-93545c8cea94" containerName="payload-processing" containerID="cri-o://96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3" gracePeriod=25
Nov 01 20:29:00.933820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:00.933762    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:29:01.057522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.057369    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\") pod \"6fc8d863-3b27-472e-b9a2-93545c8cea94\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") "
Nov 01 20:29:01.057522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.057415    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\") pod \"6fc8d863-3b27-472e-b9a2-93545c8cea94\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") "
Nov 01 20:29:01.057522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.057441    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\") pod \"6fc8d863-3b27-472e-b9a2-93545c8cea94\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") "
Nov 01 20:29:01.057861 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.057620    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\") pod \"6fc8d863-3b27-472e-b9a2-93545c8cea94\" (UID: \"6fc8d863-3b27-472e-b9a2-93545c8cea94\") "
Nov 01 20:29:01.058208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.058165    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume" pod "6fc8d863-3b27-472e-b9a2-93545c8cea94" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:01.058208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.058178    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume" pod "6fc8d863-3b27-472e-b9a2-93545c8cea94" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:01.060918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.060692    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials" pod "6fc8d863-3b27-472e-b9a2-93545c8cea94" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:01.068688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.068667    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw" pod "6fc8d863-3b27-472e-b9a2-93545c8cea94" (UID: "6fc8d863-3b27-472e-b9a2-93545c8cea94"). InnerVolumeSpecName "kube-api-access-2r8vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:01.163419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.160182    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8vw\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-kube-api-access-2r8vw\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:29:01.163419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.160235    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:29:01.163419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.160250    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8d863-3b27-472e-b9a2-93545c8cea94-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:29:01.163419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.160266    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc8d863-3b27-472e-b9a2-93545c8cea94-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:29:01.620448 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.620411    5066 generic.go:356] "Generic (PLEG): container finished" podID="6fc8d863-3b27-472e-b9a2-93545c8cea94" containerID="96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3" exitCode=0
Nov 01 20:29:01.620635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.620453    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"
Nov 01 20:29:01.620635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.620482    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" event={"ID":"6fc8d863-3b27-472e-b9a2-93545c8cea94","Type":"ContainerDied","Data":"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"}
Nov 01 20:29:01.620635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.620542    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c" event={"ID":"6fc8d863-3b27-472e-b9a2-93545c8cea94","Type":"ContainerDied","Data":"49ec3ca0426f6db4c1dc74954e38993ec78a9b9e343a86336717dc987cd017a4"}
Nov 01 20:29:01.620635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.620565    5066 scope.go:122] "RemoveContainer" containerID="96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"
Nov 01 20:29:01.631651 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.631620    5066 scope.go:122] "RemoveContainer" containerID="96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"
Nov 01 20:29:01.631917 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:29:01.631895    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3\": container with ID starting with 96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3 not found: ID does not exist" containerID="96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"
Nov 01 20:29:01.632021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.631922    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3"} err="failed to get container status \"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3\": rpc error: code = NotFound desc = could not find container \"96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3\": container with ID starting with 96b85d4d088bb5d7380cae6b89f4b015236a549a868c54398e7112867e9041e3 not found: ID does not exist"
Nov 01 20:29:01.785231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.785196    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:29:01.834905 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:01.834872    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-f74cw276c"]
Nov 01 20:29:02.731835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:29:02.731801    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc8d863-3b27-472e-b9a2-93545c8cea94" path="/var/lib/kubelet/pods/6fc8d863-3b27-472e-b9a2-93545c8cea94/volumes"
Nov 01 20:29:21.629812 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:29:21.629770    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:29:51.628908 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:29:51.628865    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:30:00.167902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.167867    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:00.168877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.168845    5066 cpu_manager.go:414] "containerMap: removing container" podUID="6fc8d863-3b27-472e-b9a2-93545c8cea94" containerName="payload-processing"
Nov 01 20:30:00.168877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.168870    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc8d863-3b27-472e-b9a2-93545c8cea94" containerName="payload-processing"
Nov 01 20:30:00.169086 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.169008    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc8d863-3b27-472e-b9a2-93545c8cea94" containerName="payload-processing"
Nov 01 20:30:00.172735 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.172715    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.189261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.189237    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:00.193340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.193316    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:00.193449 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.193417    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.198469 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.198449    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.235638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.235616    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\""
Nov 01 20:30:00.268539 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.268512    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:00.272541 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.272513    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:00.276686 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.276658    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.276816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.276750    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnq7z\" (UniqueName: \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.276816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.276790    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.276927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.276915    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxld\" (UniqueName: \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.276986 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.276976    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.277040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.277025    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzt5\" (UniqueName: \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.322202 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.322168    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:00.378663 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378601    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzt5\" (UniqueName: \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.378847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378708    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.378847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378783    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnq7z\" (UniqueName: \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.378847 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378818    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.378997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378931    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxld\" (UniqueName: \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.378997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.378985    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.379815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.379785    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.380784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.380756    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.381674 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.381651    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.436723 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.436638    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzt5\" (UniqueName: \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\") pod \"maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.437596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.437577    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnq7z\" (UniqueName: \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\") pod \"maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.439053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.439029    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxld\" (UniqueName: \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.489923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.489890    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:00.510371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.510345    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:00.517141 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.517113    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:00.887867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.887829    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:00.954217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.954063    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:00.958421 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:30:00.958384    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc58fd2_da41_4fd3_bd81_ae4d819035ab.slice/crio-4c1a42405160362214059d66653b8976ff0327d1068580373328705d46846b46 WatchSource:0}: Error finding container 4c1a42405160362214059d66653b8976ff0327d1068580373328705d46846b46: Status 404 returned error can't find the container with id 4c1a42405160362214059d66653b8976ff0327d1068580373328705d46846b46
Nov 01 20:30:00.960461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:00.960435    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:00.971164 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:30:00.971133    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc7f981_e4ab_4d55_b0a9_8d35bc4f6c62.slice/crio-d86341b2406c95293fb985d67ea9ae2a16420387976830045e2e1ce710f9b164 WatchSource:0}: Error finding container d86341b2406c95293fb985d67ea9ae2a16420387976830045e2e1ce710f9b164: Status 404 returned error can't find the container with id d86341b2406c95293fb985d67ea9ae2a16420387976830045e2e1ce710f9b164
Nov 01 20:30:01.068308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:01.068278    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerStarted","Data":"4c1a42405160362214059d66653b8976ff0327d1068580373328705d46846b46"}
Nov 01 20:30:01.069887 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:01.069860    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerStarted","Data":"d86341b2406c95293fb985d67ea9ae2a16420387976830045e2e1ce710f9b164"}
Nov 01 20:30:01.071242 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:01.071221    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerStarted","Data":"1a7c9a02171994a86e7d47e4bef5e19a08c8a407693da8be3766b80f75cfd23a"}
Nov 01 20:30:02.080536 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.080488    5066 generic.go:356] "Generic (PLEG): container finished" podID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerID="c8bc555de775f22d9e9fc6db8d0c6683f1532e25a9b86b4335115a2e3527cb15" exitCode=60
Nov 01 20:30:02.080927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.080545    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerDied","Data":"c8bc555de775f22d9e9fc6db8d0c6683f1532e25a9b86b4335115a2e3527cb15"}
Nov 01 20:30:02.080927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.080885    5066 scope.go:122] "RemoveContainer" containerID="c8bc555de775f22d9e9fc6db8d0c6683f1532e25a9b86b4335115a2e3527cb15"
Nov 01 20:30:02.082450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.082429    5066 generic.go:356] "Generic (PLEG): container finished" podID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerID="6e3548141a5a27aa75ff7aea766861e203bd981a7982c1ed8fa971e10ff10da6" exitCode=60
Nov 01 20:30:02.082589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.082483    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerDied","Data":"6e3548141a5a27aa75ff7aea766861e203bd981a7982c1ed8fa971e10ff10da6"}
Nov 01 20:30:02.082849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.082781    5066 scope.go:122] "RemoveContainer" containerID="6e3548141a5a27aa75ff7aea766861e203bd981a7982c1ed8fa971e10ff10da6"
Nov 01 20:30:02.085136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.085114    5066 generic.go:356] "Generic (PLEG): container finished" podID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerID="4de2f222adbf77f609ce4b5bcdbb3905f7e1263ed1c56ab1de4eaaec13ed3d14" exitCode=60
Nov 01 20:30:02.085237 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.085181    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerDied","Data":"4de2f222adbf77f609ce4b5bcdbb3905f7e1263ed1c56ab1de4eaaec13ed3d14"}
Nov 01 20:30:02.085414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:02.085397    5066 scope.go:122] "RemoveContainer" containerID="4de2f222adbf77f609ce4b5bcdbb3905f7e1263ed1c56ab1de4eaaec13ed3d14"
Nov 01 20:30:03.097854 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.097811    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerStarted","Data":"6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"}
Nov 01 20:30:03.100297 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.100267    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerStarted","Data":"cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"}
Nov 01 20:30:03.102629 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.102601    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerStarted","Data":"4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"}
Nov 01 20:30:03.253519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.243819    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" podStartSLOduration=3.243801796 podStartE2EDuration="3.243801796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.238391571 +0000 UTC m=+1656.623289162" watchObservedRunningTime="2026-11-01 20:30:03.243801796 +0000 UTC m=+1656.628699388"
Nov 01 20:30:03.354228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.354122    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" podStartSLOduration=3.354104055 podStartE2EDuration="3.354104055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.350640792 +0000 UTC m=+1656.735538394" watchObservedRunningTime="2026-11-01 20:30:03.354104055 +0000 UTC m=+1656.739001648"
Nov 01 20:30:03.463528 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:03.463441    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" podStartSLOduration=3.46342565 podStartE2EDuration="3.46342565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.461619696 +0000 UTC m=+1656.846517289" watchObservedRunningTime="2026-11-01 20:30:03.46342565 +0000 UTC m=+1656.848323242"
Nov 01 20:30:04.111879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.111839    5066 generic.go:356] "Generic (PLEG): container finished" podID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerID="4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5" exitCode=60
Nov 01 20:30:04.112371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.111925    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerDied","Data":"4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"}
Nov 01 20:30:04.112371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.111997    5066 scope.go:122] "RemoveContainer" containerID="c8bc555de775f22d9e9fc6db8d0c6683f1532e25a9b86b4335115a2e3527cb15"
Nov 01 20:30:04.112371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.112220    5066 scope.go:122] "RemoveContainer" containerID="4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"
Nov 01 20:30:04.112589 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:04.112549    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh_odh-ai-gateway-infra(cfc58fd2-da41-4fd3-bd81-ae4d819035ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab"
Nov 01 20:30:04.113929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.113903    5066 generic.go:356] "Generic (PLEG): container finished" podID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerID="6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b" exitCode=60
Nov 01 20:30:04.114050 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.113955    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerDied","Data":"6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"}
Nov 01 20:30:04.114256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.114238    5066 scope.go:122] "RemoveContainer" containerID="6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"
Nov 01 20:30:04.114544 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:04.114525    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw_odh-ai-gateway-infra(ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62"
Nov 01 20:30:04.115942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.115920    5066 generic.go:356] "Generic (PLEG): container finished" podID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerID="cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7" exitCode=60
Nov 01 20:30:04.116048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.115971    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerDied","Data":"cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"}
Nov 01 20:30:04.116199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.116186    5066 scope.go:122] "RemoveContainer" containerID="cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"
Nov 01 20:30:04.116442 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:04.116423    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9_odh-ai-gateway-infra(4b1779a3-78dc-410f-a72a-907ea521ffd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5"
Nov 01 20:30:04.124902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.124872    5066 scope.go:122] "RemoveContainer" containerID="6e3548141a5a27aa75ff7aea766861e203bd981a7982c1ed8fa971e10ff10da6"
Nov 01 20:30:04.137761 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:04.137745    5066 scope.go:122] "RemoveContainer" containerID="4de2f222adbf77f609ce4b5bcdbb3905f7e1263ed1c56ab1de4eaaec13ed3d14"
Nov 01 20:30:05.124053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:05.124015    5066 scope.go:122] "RemoveContainer" containerID="6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"
Nov 01 20:30:05.124543 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:05.124334    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw_odh-ai-gateway-infra(ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62"
Nov 01 20:30:05.125846 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:05.125824    5066 scope.go:122] "RemoveContainer" containerID="cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"
Nov 01 20:30:05.126125 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:05.126105    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9_odh-ai-gateway-infra(4b1779a3-78dc-410f-a72a-907ea521ffd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5"
Nov 01 20:30:05.127552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:05.127534    5066 scope.go:122] "RemoveContainer" containerID="4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"
Nov 01 20:30:05.127776 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:05.127755    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh_odh-ai-gateway-infra(cfc58fd2-da41-4fd3-bd81-ae4d819035ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab"
Nov 01 20:30:09.223747 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:09.223666    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:30:09.332530 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:09.332465    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:30:14.224766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.224713    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" podUID="653918fe-ba45-4bbe-b860-60b3fb361de9" containerName="payload-processing" containerID="cri-o://1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290" gracePeriod=25
Nov 01 20:30:14.333078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.333036    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" podUID="0b13025e-eeb6-4d80-815d-765ae432cb88" containerName="payload-pre-processing" containerID="cri-o://ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d" gracePeriod=25
Nov 01 20:30:14.535196 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.535165    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:30:14.609654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.609622    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:30:14.691252 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.691221    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\") pod \"653918fe-ba45-4bbe-b860-60b3fb361de9\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") "
Nov 01 20:30:14.691434 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.691279    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\" (UniqueName: \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\") pod \"653918fe-ba45-4bbe-b860-60b3fb361de9\" (UID: \"653918fe-ba45-4bbe-b860-60b3fb361de9\") "
Nov 01 20:30:14.691718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.691689    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume" pod "653918fe-ba45-4bbe-b860-60b3fb361de9" (UID: "653918fe-ba45-4bbe-b860-60b3fb361de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:14.691856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.691841    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653918fe-ba45-4bbe-b860-60b3fb361de9-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:14.693442 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.693420    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj" pod "653918fe-ba45-4bbe-b860-60b3fb361de9" (UID: "653918fe-ba45-4bbe-b860-60b3fb361de9"). InnerVolumeSpecName "kube-api-access-bnjnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:14.729370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.726105    5066 scope.go:122] "RemoveContainer" containerID="cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"
Nov 01 20:30:14.729370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.726515    5066 scope.go:122] "RemoveContainer" containerID="6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"
Nov 01 20:30:14.792325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.792294    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\") pod \"0b13025e-eeb6-4d80-815d-765ae432cb88\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") "
Nov 01 20:30:14.792435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.792424    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\" (UniqueName: \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\") pod \"0b13025e-eeb6-4d80-815d-765ae432cb88\" (UID: \"0b13025e-eeb6-4d80-815d-765ae432cb88\") "
Nov 01 20:30:14.792784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.792756    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume" pod "0b13025e-eeb6-4d80-815d-765ae432cb88" (UID: "0b13025e-eeb6-4d80-815d-765ae432cb88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:14.792916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.792896    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjnj\" (UniqueName: \"kubernetes.io/projected/653918fe-ba45-4bbe-b860-60b3fb361de9-kube-api-access-bnjnj\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:14.792981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.792920    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b13025e-eeb6-4d80-815d-765ae432cb88-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:14.794876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.794848    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs" pod "0b13025e-eeb6-4d80-815d-765ae432cb88" (UID: "0b13025e-eeb6-4d80-815d-765ae432cb88"). InnerVolumeSpecName "kube-api-access-brmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:14.894434 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:14.894399    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmfs\" (UniqueName: \"kubernetes.io/projected/0b13025e-eeb6-4d80-815d-765ae432cb88-kube-api-access-brmfs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:15.198015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.197986    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerStarted","Data":"ff81affba2ee6c631a38f74f86d3a0f29ef24bec35dea2e84ff655fbac5349a4"}
Nov 01 20:30:15.200145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.200119    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerStarted","Data":"6d5899f22e193337f8d4fc533078004bc7c4d043d2571d8854d58c344ccae5c9"}
Nov 01 20:30:15.201704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.201674    5066 generic.go:356] "Generic (PLEG): container finished" podID="0b13025e-eeb6-4d80-815d-765ae432cb88" containerID="ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d" exitCode=0
Nov 01 20:30:15.201815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.201703    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" event={"ID":"0b13025e-eeb6-4d80-815d-765ae432cb88","Type":"ContainerDied","Data":"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"}
Nov 01 20:30:15.201815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.201732    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"
Nov 01 20:30:15.201815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.201759    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj" event={"ID":"0b13025e-eeb6-4d80-815d-765ae432cb88","Type":"ContainerDied","Data":"727b827a5dbb94e79dd3d056b4b341de08c5b22ba73f5425dcaa6ccaaeee6508"}
Nov 01 20:30:15.201815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.201789    5066 scope.go:122] "RemoveContainer" containerID="ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"
Nov 01 20:30:15.203261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.203238    5066 generic.go:356] "Generic (PLEG): container finished" podID="653918fe-ba45-4bbe-b860-60b3fb361de9" containerID="1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290" exitCode=0
Nov 01 20:30:15.203261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.203270    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" event={"ID":"653918fe-ba45-4bbe-b860-60b3fb361de9","Type":"ContainerDied","Data":"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"}
Nov 01 20:30:15.203407 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.203291    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"
Nov 01 20:30:15.203407 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.203304    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz" event={"ID":"653918fe-ba45-4bbe-b860-60b3fb361de9","Type":"ContainerDied","Data":"520ae0e553c3ff09241e2a8ee9ba0c5169b892fcdda98e4c3271b1352a8d0ad5"}
Nov 01 20:30:15.212467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.212443    5066 scope.go:122] "RemoveContainer" containerID="ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"
Nov 01 20:30:15.212813 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:15.212795    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d\": container with ID starting with ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d not found: ID does not exist" containerID="ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"
Nov 01 20:30:15.212881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.212819    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d"} err="failed to get container status \"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d\": rpc error: code = NotFound desc = could not find container \"ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d\": container with ID starting with ca06a5d33a5940abcdf27086c915b76c88862c035c348fe5090cd357b9c80b8d not found: ID does not exist"
Nov 01 20:30:15.212881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.212835    5066 scope.go:122] "RemoveContainer" containerID="1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"
Nov 01 20:30:15.222625 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.222607    5066 scope.go:122] "RemoveContainer" containerID="1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"
Nov 01 20:30:15.222889 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:15.222871    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290\": container with ID starting with 1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290 not found: ID does not exist" containerID="1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"
Nov 01 20:30:15.222929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.222899    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290"} err="failed to get container status \"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290\": rpc error: code = NotFound desc = could not find container \"1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290\": container with ID starting with 1e6f90b83065622a1531a5811c6dda4cb466bef25c8722e1324c52da0fc6b290 not found: ID does not exist"
Nov 01 20:30:15.536045 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.536007    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:30:15.540671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.540642    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7387069c-75bc8876f-6m6rz"]
Nov 01 20:30:15.652622 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.652584    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:30:15.655844 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:15.655813    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7387069c-57bd96f57f-tj6zj"]
Nov 01 20:30:16.213906 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.213873    5066 generic.go:356] "Generic (PLEG): container finished" podID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerID="ff81affba2ee6c631a38f74f86d3a0f29ef24bec35dea2e84ff655fbac5349a4" exitCode=60
Nov 01 20:30:16.214107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.213946    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerDied","Data":"ff81affba2ee6c631a38f74f86d3a0f29ef24bec35dea2e84ff655fbac5349a4"}
Nov 01 20:30:16.214107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.214044    5066 scope.go:122] "RemoveContainer" containerID="6f496f8d5a1bde5000b6c7d9b4bd1fd6cc3f3976a6a6f4491f8c03217079902b"
Nov 01 20:30:16.214435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.214411    5066 scope.go:122] "RemoveContainer" containerID="ff81affba2ee6c631a38f74f86d3a0f29ef24bec35dea2e84ff655fbac5349a4"
Nov 01 20:30:16.214764 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:16.214739    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw_odh-ai-gateway-infra(ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62"
Nov 01 20:30:16.216300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.216278    5066 generic.go:356] "Generic (PLEG): container finished" podID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerID="6d5899f22e193337f8d4fc533078004bc7c4d043d2571d8854d58c344ccae5c9" exitCode=60
Nov 01 20:30:16.216375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.216355    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerDied","Data":"6d5899f22e193337f8d4fc533078004bc7c4d043d2571d8854d58c344ccae5c9"}
Nov 01 20:30:16.216764 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.216748    5066 scope.go:122] "RemoveContainer" containerID="6d5899f22e193337f8d4fc533078004bc7c4d043d2571d8854d58c344ccae5c9"
Nov 01 20:30:16.217018 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:16.216998    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9_odh-ai-gateway-infra(4b1779a3-78dc-410f-a72a-907ea521ffd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5"
Nov 01 20:30:16.226851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.226817    5066 scope.go:122] "RemoveContainer" containerID="cd10c1ff773f1d9381569f2a9a4f0e285fbe525a75f18592ffb0389136842da7"
Nov 01 20:30:16.321422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.321388    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:16.432244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.432213    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:16.729410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.729379    5066 scope.go:122] "RemoveContainer" containerID="4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"
Nov 01 20:30:16.734007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.733980    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b13025e-eeb6-4d80-815d-765ae432cb88" path="/var/lib/kubelet/pods/0b13025e-eeb6-4d80-815d-765ae432cb88/volumes"
Nov 01 20:30:16.734476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:16.734456    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653918fe-ba45-4bbe-b860-60b3fb361de9" path="/var/lib/kubelet/pods/653918fe-ba45-4bbe-b860-60b3fb361de9/volumes"
Nov 01 20:30:17.228018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.227937    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerStarted","Data":"effc9df5b1bfa637395cfde070ed9e4b8f1673f357ab535ca31b5e2d0ebe7802"}
Nov 01 20:30:17.471208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.471179    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:17.513644 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.513617    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:17.527969 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.527941    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\" (UniqueName: \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\") pod \"4b1779a3-78dc-410f-a72a-907ea521ffd5\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") "
Nov 01 20:30:17.528112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.527990    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\") pod \"4b1779a3-78dc-410f-a72a-907ea521ffd5\" (UID: \"4b1779a3-78dc-410f-a72a-907ea521ffd5\") "
Nov 01 20:30:17.528179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528159    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\") pod \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") "
Nov 01 20:30:17.528251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528238    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\" (UniqueName: \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\") pod \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\" (UID: \"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62\") "
Nov 01 20:30:17.528313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528295    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca" pod "4b1779a3-78dc-410f-a72a-907ea521ffd5" (UID: "4b1779a3-78dc-410f-a72a-907ea521ffd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:17.528505 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528477    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca" pod "ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" (UID: "ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:17.528858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528841    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1779a3-78dc-410f-a72a-907ea521ffd5-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.528910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.528860    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.530125 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.530100    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld" pod "4b1779a3-78dc-410f-a72a-907ea521ffd5" (UID: "4b1779a3-78dc-410f-a72a-907ea521ffd5"). InnerVolumeSpecName "kube-api-access-6gxld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:17.530292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.530276    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5" pod "ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" (UID: "ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62"). InnerVolumeSpecName "kube-api-access-pvzt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:17.630556 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.630525    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzt5\" (UniqueName: \"kubernetes.io/projected/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62-kube-api-access-pvzt5\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.630556 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:17.630552    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxld\" (UniqueName: \"kubernetes.io/projected/4b1779a3-78dc-410f-a72a-907ea521ffd5-kube-api-access-6gxld\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.245142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.245115    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"
Nov 01 20:30:18.245605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.245149    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw" event={"ID":"ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62","Type":"ContainerDied","Data":"d86341b2406c95293fb985d67ea9ae2a16420387976830045e2e1ce710f9b164"}
Nov 01 20:30:18.245605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.245233    5066 scope.go:122] "RemoveContainer" containerID="ff81affba2ee6c631a38f74f86d3a0f29ef24bec35dea2e84ff655fbac5349a4"
Nov 01 20:30:18.246590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.246569    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"
Nov 01 20:30:18.246677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.246611    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9" event={"ID":"4b1779a3-78dc-410f-a72a-907ea521ffd5","Type":"ContainerDied","Data":"1a7c9a02171994a86e7d47e4bef5e19a08c8a407693da8be3766b80f75cfd23a"}
Nov 01 20:30:18.248920 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.248894    5066 generic.go:356] "Generic (PLEG): container finished" podID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerID="effc9df5b1bfa637395cfde070ed9e4b8f1673f357ab535ca31b5e2d0ebe7802" exitCode=60
Nov 01 20:30:18.249024 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.248938    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerDied","Data":"effc9df5b1bfa637395cfde070ed9e4b8f1673f357ab535ca31b5e2d0ebe7802"}
Nov 01 20:30:18.249316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.249298    5066 scope.go:122] "RemoveContainer" containerID="effc9df5b1bfa637395cfde070ed9e4b8f1673f357ab535ca31b5e2d0ebe7802"
Nov 01 20:30:18.249631 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:18.249602    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh_odh-ai-gateway-infra(cfc58fd2-da41-4fd3-bd81-ae4d819035ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab"
Nov 01 20:30:18.255440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.255416    5066 scope.go:122] "RemoveContainer" containerID="6d5899f22e193337f8d4fc533078004bc7c4d043d2571d8854d58c344ccae5c9"
Nov 01 20:30:18.264603 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.264585    5066 scope.go:122] "RemoveContainer" containerID="4d291991312aa6a29e2dd9a1fa80a81f1c4dd62f93669f1a1306cc090dcd68a5"
Nov 01 20:30:18.359052 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.359013    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:18.416597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.416565    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:18.419849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.419822    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892750-lmkh9"]
Nov 01 20:30:18.524128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.524043    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:18.530862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.530832    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-1b1134-29892750-6cvpw"]
Nov 01 20:30:18.733889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.733854    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" path="/var/lib/kubelet/pods/4b1779a3-78dc-410f-a72a-907ea521ffd5/volumes"
Nov 01 20:30:18.734280 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:18.734267    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" path="/var/lib/kubelet/pods/ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62/volumes"
Nov 01 20:30:19.473756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.473722    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:19.561766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.561673    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\" (UniqueName: \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\") pod \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") "
Nov 01 20:30:19.561766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.561739    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\") pod \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\" (UID: \"cfc58fd2-da41-4fd3-bd81-ae4d819035ab\") "
Nov 01 20:30:19.562143 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.562105    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca" pod "cfc58fd2-da41-4fd3-bd81-ae4d819035ab" (UID: "cfc58fd2-da41-4fd3-bd81-ae4d819035ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:19.562383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.562359    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:19.564071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.564040    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z" pod "cfc58fd2-da41-4fd3-bd81-ae4d819035ab" (UID: "cfc58fd2-da41-4fd3-bd81-ae4d819035ab"). InnerVolumeSpecName "kube-api-access-vnq7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:19.664003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:19.663958    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnq7z\" (UniqueName: \"kubernetes.io/projected/cfc58fd2-da41-4fd3-bd81-ae4d819035ab-kube-api-access-vnq7z\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.270237 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.270203    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"
Nov 01 20:30:20.270425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.270237    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh" event={"ID":"cfc58fd2-da41-4fd3-bd81-ae4d819035ab","Type":"ContainerDied","Data":"4c1a42405160362214059d66653b8976ff0327d1068580373328705d46846b46"}
Nov 01 20:30:20.270425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.270300    5066 scope.go:122] "RemoveContainer" containerID="effc9df5b1bfa637395cfde070ed9e4b8f1673f357ab535ca31b5e2d0ebe7802"
Nov 01 20:30:20.448512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.448459    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:20.452414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.452381    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d204f4-29892750-zgzrh"]
Nov 01 20:30:20.733525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:20.733478    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" path="/var/lib/kubelet/pods/cfc58fd2-da41-4fd3-bd81-ae4d819035ab/volumes"
Nov 01 20:30:21.630238 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:21.630199    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:30:43.463077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:43.463041    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:30:43.570288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:43.570236    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:30:48.465523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:48.465462    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerName="payload-processing" containerID="cri-o://76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d" gracePeriod=25
Nov 01 20:30:48.571612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:48.571571    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" podUID="cf734fab-ca0d-4c51-97f9-32d451686b3b" containerName="payload-pre-processing" containerID="cri-o://9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b" gracePeriod=25
Nov 01 20:30:49.039591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.039556    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:30:49.058929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.058891    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\") pod \"cf734fab-ca0d-4c51-97f9-32d451686b3b\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") "
Nov 01 20:30:49.059109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.059014    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\" (UniqueName: \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\") pod \"cf734fab-ca0d-4c51-97f9-32d451686b3b\" (UID: \"cf734fab-ca0d-4c51-97f9-32d451686b3b\") "
Nov 01 20:30:49.059318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.059291    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume" pod "cf734fab-ca0d-4c51-97f9-32d451686b3b" (UID: "cf734fab-ca0d-4c51-97f9-32d451686b3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:49.059587 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.059568    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf734fab-ca0d-4c51-97f9-32d451686b3b-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:49.123571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.123510    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h" pod "cf734fab-ca0d-4c51-97f9-32d451686b3b" (UID: "cf734fab-ca0d-4c51-97f9-32d451686b3b"). InnerVolumeSpecName "kube-api-access-nkq8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:49.144091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.144061    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:30:49.159918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.159884    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\") pod \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") "
Nov 01 20:30:49.160120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.160097    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\" (UniqueName: \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\") pod \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\" (UID: \"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c\") "
Nov 01 20:30:49.160270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.160247    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume" pod "dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" (UID: "dd2b2186-fe31-4c83-a0aa-60a28a04fd1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:49.160668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.160645    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq8h\" (UniqueName: \"kubernetes.io/projected/cf734fab-ca0d-4c51-97f9-32d451686b3b-kube-api-access-nkq8h\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:49.160668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.160667    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:49.162261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.162235    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww" pod "dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" (UID: "dd2b2186-fe31-4c83-a0aa-60a28a04fd1c"). InnerVolumeSpecName "kube-api-access-jk5ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:49.262208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.262172    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5ww\" (UniqueName: \"kubernetes.io/projected/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c-kube-api-access-jk5ww\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:30:49.514670 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.514636    5066 generic.go:356] "Generic (PLEG): container finished" podID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerID="76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d" exitCode=0
Nov 01 20:30:49.515142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.514680    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"
Nov 01 20:30:49.515142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.514718    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" event={"ID":"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c","Type":"ContainerDied","Data":"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"}
Nov 01 20:30:49.515142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.514783    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" event={"ID":"dd2b2186-fe31-4c83-a0aa-60a28a04fd1c","Type":"ContainerDied","Data":"e68de1248a17957f69ff0c35a0a042b9fae3fae0fc49053ab767e1165c82c82a"}
Nov 01 20:30:49.515142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.514813    5066 scope.go:122] "RemoveContainer" containerID="76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"
Nov 01 20:30:49.516352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.516224    5066 generic.go:356] "Generic (PLEG): container finished" podID="cf734fab-ca0d-4c51-97f9-32d451686b3b" containerID="9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b" exitCode=0
Nov 01 20:30:49.516352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.516321    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" event={"ID":"cf734fab-ca0d-4c51-97f9-32d451686b3b","Type":"ContainerDied","Data":"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"}
Nov 01 20:30:49.516447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.516360    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj" event={"ID":"cf734fab-ca0d-4c51-97f9-32d451686b3b","Type":"ContainerDied","Data":"0dc9b5fbd2271a00e06157f0a6883a85bfe3d64d7375d1bb1e3f90ae4c85a7e0"}
Nov 01 20:30:49.516447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.516428    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"
Nov 01 20:30:49.525826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.525805    5066 scope.go:122] "RemoveContainer" containerID="76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"
Nov 01 20:30:49.526166 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:49.526142    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d\": container with ID starting with 76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d not found: ID does not exist" containerID="76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"
Nov 01 20:30:49.526265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.526171    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d"} err="failed to get container status \"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d\": rpc error: code = NotFound desc = could not find container \"76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d\": container with ID starting with 76bb512b33ea5da760ebaa2dc16bb49cce466f59e4a62057cf345fe5564ccd6d not found: ID does not exist"
Nov 01 20:30:49.526265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.526189    5066 scope.go:122] "RemoveContainer" containerID="9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"
Nov 01 20:30:49.536160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.536134    5066 scope.go:122] "RemoveContainer" containerID="9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"
Nov 01 20:30:49.536454 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:49.536434    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b\": container with ID starting with 9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b not found: ID does not exist" containerID="9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"
Nov 01 20:30:49.536526 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.536464    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b"} err="failed to get container status \"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b\": rpc error: code = NotFound desc = could not find container \"9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b\": container with ID starting with 9c2e3b9755538dbc4141d9533acf7b0d37b6228864ce1bec284a9145d9a3ce9b not found: ID does not exist"
Nov 01 20:30:49.679111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.679073    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:30:49.682802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.682774    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k"]
Nov 01 20:30:49.790770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.790173    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:30:49.793875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:49.793850    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-202d23-5b87fb576c-4q2dj"]
Nov 01 20:30:50.027578 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:50.027533    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-202d23-857bc57847-vct2k" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.110:50052\" within 1s: context deadline exceeded"
Nov 01 20:30:50.027761 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:30:50.027651    5066 logging.go:55] [core] [Channel #3731 SubChannel #3732]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.110:50052", ServerName: "10.132.0.110:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.110:50052: operation was canceled"
Nov 01 20:30:50.732712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:50.732676    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf734fab-ca0d-4c51-97f9-32d451686b3b" path="/var/lib/kubelet/pods/cf734fab-ca0d-4c51-97f9-32d451686b3b/volumes"
Nov 01 20:30:50.733142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:30:50.733127    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" path="/var/lib/kubelet/pods/dd2b2186-fe31-4c83-a0aa-60a28a04fd1c/volumes"
Nov 01 20:30:51.668625 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:30:51.668452    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:31:03.352389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352351    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:31:03.352945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352930    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0b13025e-eeb6-4d80-815d-765ae432cb88" containerName="payload-pre-processing"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352946    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b13025e-eeb6-4d80-815d-765ae432cb88" containerName="payload-pre-processing"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352959    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352964    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352973    5066 cpu_manager.go:414] "containerMap: removing container" podUID="653918fe-ba45-4bbe-b860-60b3fb361de9" containerName="payload-processing"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352979    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653918fe-ba45-4bbe-b860-60b3fb361de9" containerName="payload-processing"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352992    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.352996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.352997    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353009    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353015    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353032    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cf734fab-ca0d-4c51-97f9-32d451686b3b" containerName="payload-pre-processing"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353039    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf734fab-ca0d-4c51-97f9-32d451686b3b" containerName="payload-pre-processing"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353054    5066 cpu_manager.go:414] "containerMap: removing container" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerName="payload-processing"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353059    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerName="payload-processing"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353072    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353077    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353084    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353088    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353099    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353104    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353112    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353116    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353122    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353125    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353214    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353221    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf734fab-ca0d-4c51-97f9-32d451686b3b" containerName="payload-pre-processing"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353228    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353234    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2b2186-fe31-4c83-a0aa-60a28a04fd1c" containerName="payload-processing"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353241    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353248    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353253    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc7f981-e4ab-4d55-b0a9-8d35bc4f6c62" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353261    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b13025e-eeb6-4d80-815d-765ae432cb88" containerName="payload-pre-processing"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353266    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353271    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="653918fe-ba45-4bbe-b860-60b3fb361de9" containerName="payload-processing"
Nov 01 20:31:03.353722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.353277    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:03.358040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.358020    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442036 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442003    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442036 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442040    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442168    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442216    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442294    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjtq\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442354 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442346    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442400    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442524    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.442607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.442545    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.464582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.464549    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:31:03.544009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.543958    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544053    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544163    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544187    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544220    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544249    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544470 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544441    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544547    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544613    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544697 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544637    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjtq\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544837    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544860    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544880    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.544999 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.544920    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.547013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.546983    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.547232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.547210    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.601510 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.601465    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.601645 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.601612    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjtq\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:03.672017 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:03.671923    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:04.103471 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:31:04.103439    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859b2536_9662_47b2_a4bc_cdd1dc9e7e78.slice/crio-e4c76153f3e231b561fddd9babb3bf091ca40053ec798ec44f48463a90245a48 WatchSource:0}: Error finding container e4c76153f3e231b561fddd9babb3bf091ca40053ec798ec44f48463a90245a48: Status 404 returned error can't find the container with id e4c76153f3e231b561fddd9babb3bf091ca40053ec798ec44f48463a90245a48
Nov 01 20:31:04.103826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.103798    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:31:04.106104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.106071    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:04.106204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.106140    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:04.106204 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.106168    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:04.641097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.640228    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" event={"ID":"859b2536-9662-47b2-a4bc-cdd1dc9e7e78","Type":"ContainerStarted","Data":"22fa1c5179d2a2c45bd0f17df30306269cad030e2ba08ff9c617e44066d2504f"}
Nov 01 20:31:04.641097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.640356    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" event={"ID":"859b2536-9662-47b2-a4bc-cdd1dc9e7e78","Type":"ContainerStarted","Data":"e4c76153f3e231b561fddd9babb3bf091ca40053ec798ec44f48463a90245a48"}
Nov 01 20:31:04.672369 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.672336    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:04.754471 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:04.754409    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podStartSLOduration=1.754392873 podStartE2EDuration="1.754392873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:04.754046758 +0000 UTC m=+1718.138944348" watchObservedRunningTime="2026-11-01 20:31:04.754392873 +0000 UTC m=+1718.139290464"
Nov 01 20:31:05.688516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:05.687633    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:05.688516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:05.687701    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:06.685529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:06.682590    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:06.685529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:06.682661    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:07.681518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:07.675948    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:07.681518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:07.676036    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:08.692356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:08.682612    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:08.692356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:08.682696    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:09.682519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:09.679925    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:09.682519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:09.680008    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:10.672775 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:10.672742    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 20:31:10.673183 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:10.672803    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 20:31:11.676731 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:11.676701    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:11.677437 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:11.677387    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:11.678312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:11.678295    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:31:18.743884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.743849    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:31:18.744486 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.744470    5066 cpu_manager.go:414] "containerMap: removing container" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:18.744555 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.744487    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:18.744634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.744625    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1779a3-78dc-410f-a72a-907ea521ffd5" containerName="cleanup"
Nov 01 20:31:18.744668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.744637    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc58fd2-da41-4fd3-bd81-ae4d819035ab" containerName="cleanup"
Nov 01 20:31:18.748138 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.748117    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.768969    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjg87\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769139 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769029    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769289 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769255    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769363    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769598 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769531    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769623    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769650    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769688 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769677    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.769762 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.769704    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.800413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.800371    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71335882-openshift-default-dockercfg-f4t24\""
Nov 01 20:31:18.854909 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.854870    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:31:18.871009 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.870974    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjg87\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871031    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871090    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871138    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871243    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871300    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871326    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871352    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871377    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.871840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.871817    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.872108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.872088    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.872809 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.872507    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.872971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.872590    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.872971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.872744    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.875164 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.875141    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.877363 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.877340    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.929508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.929458    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:18.929658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:18.929474    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjg87\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-br4r9\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:19.063412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.063323    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:19.445268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.445188    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:31:19.605874 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.605838    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:31:19.613971 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:31:19.613941    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ea3fef_03db_4c45_9a05_5cea65b76618.slice/crio-d7594900041cc962c26adb88018ae5cf368e12d8ff563775d4c24ff4ec9ffb05 WatchSource:0}: Error finding container d7594900041cc962c26adb88018ae5cf368e12d8ff563775d4c24ff4ec9ffb05: Status 404 returned error can't find the container with id d7594900041cc962c26adb88018ae5cf368e12d8ff563775d4c24ff4ec9ffb05
Nov 01 20:31:19.616408 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.616375    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:19.616518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.616443    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:19.616518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.616473    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:31:19.768227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.768187    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" event={"ID":"77ea3fef-03db-4c45-9a05-5cea65b76618","Type":"ContainerStarted","Data":"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"}
Nov 01 20:31:19.768737 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.768250    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" event={"ID":"77ea3fef-03db-4c45-9a05-5cea65b76618","Type":"ContainerStarted","Data":"d7594900041cc962c26adb88018ae5cf368e12d8ff563775d4c24ff4ec9ffb05"}
Nov 01 20:31:19.887980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:19.887926    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podStartSLOduration=1.8879099689999999 podStartE2EDuration="1.887909969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:19.88452287 +0000 UTC m=+1733.269420463" watchObservedRunningTime="2026-11-01 20:31:19.887909969 +0000 UTC m=+1733.272807560"
Nov 01 20:31:20.064656 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:20.064462    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:20.066352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:20.066325    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:20.066516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:20.066377    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:21.080518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:21.066924    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:21.080518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:21.066989    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:21.631052 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:21.631015    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:31:22.078519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:22.070898    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:22.078519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:22.070964    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:23.077518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:23.072483    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:23.077518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:23.072561    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:24.070522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:24.065921    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:24.070522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:24.065988    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:25.085519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:25.084956    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 20:31:25.085519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:25.085022    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 20:31:26.260279 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:26.260234    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:26.260812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:26.260754    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:26.261621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:26.261594    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:31:33.876440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:33.876402    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:31:33.882118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:33.882060    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.019256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019214    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.019578 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019555    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019761    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019814    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019861    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7zx\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019914    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.019949    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.020000    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.020799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.020041    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.085002 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.084970    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:31:34.154705 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.128119    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.156518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.156471    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.157142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.157119    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.157855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.157835    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7zx\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.158269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.158249    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.158407 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.158389    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.158547 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.158531    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.158679 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.158664    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.158830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.158804    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.159668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.159648    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.160036 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.160017    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.160389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.160370    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.160522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.134407    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.161253 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.161233    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.183690 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.183624    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.213520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.209191    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.220879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.220307    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7zx\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.220879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.220830    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.466441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.466377    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" podUID="15ba9911-cef5-40aa-9351-203bfe11a828" containerName="main" containerID="cri-o://2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d" gracePeriod=45
Nov 01 20:31:34.499945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.499915    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:34.722712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.722650    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:31:34.766487 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766456    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766526    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766558    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766593    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\" (UniqueName: \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766661    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766727    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:34.766911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766738    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\" (UniqueName: \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\") pod \"15ba9911-cef5-40aa-9351-203bfe11a828\" (UID: \"15ba9911-cef5-40aa-9351-203bfe11a828\") "
Nov 01 20:31:34.766911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766852    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:34.767038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.766962    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:34.767582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.767556    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.767727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.767585    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.767727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.767601    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.768936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.768909    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:34.769030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.768948    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:34.769030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.768971    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77" pod "15ba9911-cef5-40aa-9351-203bfe11a828" (UID: "15ba9911-cef5-40aa-9351-203bfe11a828"). InnerVolumeSpecName "kube-api-access-vqp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:34.868936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.868898    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ba9911-cef5-40aa-9351-203bfe11a828-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.868936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.868931    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ba9911-cef5-40aa-9351-203bfe11a828-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.869124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.868944    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqp77\" (UniqueName: \"kubernetes.io/projected/15ba9911-cef5-40aa-9351-203bfe11a828-kube-api-access-vqp77\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:31:34.899945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.899914    5066 generic.go:356] "Generic (PLEG): container finished" podID="15ba9911-cef5-40aa-9351-203bfe11a828" containerID="2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d" exitCode=0
Nov 01 20:31:34.900440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.899963    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"
Nov 01 20:31:34.900440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.900000    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" event={"ID":"15ba9911-cef5-40aa-9351-203bfe11a828","Type":"ContainerDied","Data":"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"}
Nov 01 20:31:34.900440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.900050    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk" event={"ID":"15ba9911-cef5-40aa-9351-203bfe11a828","Type":"ContainerDied","Data":"e2fe814304bc28d6e94cc1db4aa032c5b802500937db2f29f73243c7f25c122b"}
Nov 01 20:31:34.900440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.900075    5066 scope.go:122] "RemoveContainer" containerID="2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"
Nov 01 20:31:34.911730 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.911712    5066 scope.go:122] "RemoveContainer" containerID="2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"
Nov 01 20:31:34.912015 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:34.911997    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d\": container with ID starting with 2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d not found: ID does not exist" containerID="2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"
Nov 01 20:31:34.912064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:34.912024    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d"} err="failed to get container status \"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d\": rpc error: code = NotFound desc = could not find container \"2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d\": container with ID starting with 2bd936b89b0707d3344faf2ddf9b1e50be623f19089c78c110d19d6c2841ca7d not found: ID does not exist"
Nov 01 20:31:35.072480 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:35.072401    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:31:35.077570 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:35.077541    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-71335882/e2e-lifecycle-model-71335882-kserve-6c58784bd9-wmrwk"]
Nov 01 20:31:36.732071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:36.732035    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ba9911-cef5-40aa-9351-203bfe11a828" path="/var/lib/kubelet/pods/15ba9911-cef5-40aa-9351-203bfe11a828/volumes"
Nov 01 20:31:37.100122 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:37.100091    5066 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 20:31:37.100122 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f" Netns:"/var/run/netns/1440e628-1d1d-4872-84c2-18396ba40b5b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:37.100122 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:37.100122 ip-10-0-142-223 kubenswrapper[5066]:  >
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:37.100146    5066 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f" Netns:"/var/run/netns/1440e628-1d1d-4872-84c2-18396ba40b5b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:37.100164    5066 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f" Netns:"/var/run/netns/1440e628-1d1d-4872-84c2-18396ba40b5b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:37.100375 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:37.100230    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f\\\" Netns:\\\"/var/run/netns/1440e628-1d1d-4872-84c2-18396ba40b5b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=bca00e785aaffe336c0907db878f84035fdc674a88ba2908977a6d4b2e54f49f;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get \\\"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8"
Nov 01 20:31:37.920328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:37.920244    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:37.920806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:37.920664    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:40.521646 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:40.521609    5066 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 20:31:40.521646 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9" Netns:"/var/run/netns/204923ec-29e2-4538-b3b4-8b541e782993" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:40.521646 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:40.521646 ip-10-0-142-223 kubenswrapper[5066]:  >
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:40.521664    5066 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9" Netns:"/var/run/netns/204923ec-29e2-4538-b3b4-8b541e782993" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:40.521684    5066 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9" Netns:"/var/run/netns/204923ec-29e2-4538-b3b4-8b541e782993" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:40.522190 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:40.521773    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9\\\" Netns:\\\"/var/run/netns/204923ec-29e2-4538-b3b4-8b541e782993\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=012504269e8332d4705ed27201a5dc1bbec4889ef5fa443db9c3b4980314bda9;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get \\\"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8"
Nov 01 20:31:51.629059 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:51.629015    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:31:53.723467 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:53.723438    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:53.723867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:31:53.723779    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:56.320637 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:56.320595    5066 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 20:31:56.320637 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a" Netns:"/var/run/netns/8973511b-6fbc-4e33-a940-0c0ebd05bb12" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:56.320637 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:56.320637 ip-10-0-142-223 kubenswrapper[5066]:  >
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:56.320663    5066 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a" Netns:"/var/run/netns/8973511b-6fbc-4e33-a940-0c0ebd05bb12" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:56.320686    5066 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a" Netns:"/var/run/netns/8973511b-6fbc-4e33-a940-0c0ebd05bb12" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:31:56.321166 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:31:56.320804    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a\\\" Netns:\\\"/var/run/netns/8973511b-6fbc-4e33-a940-0c0ebd05bb12\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=4b75f75660b10c10261e9873a7e6f2ba67674d378cd1d2812e55848fae7ca76a;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get \\\"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8"
Nov 01 20:32:07.723571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:07.723534    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:07.724409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:07.723882    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:10.319519 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:10.319467    5066 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 20:32:10.319519 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee" Netns:"/var/run/netns/da84d06e-c29e-4fd3-bc61-4ace94ffe2ea" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:32:10.319519 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:32:10.319519 ip-10-0-142-223 kubenswrapper[5066]:  >
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:10.319545    5066 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee" Netns:"/var/run/netns/da84d06e-c29e-4fd3-bc61-4ace94ffe2ea" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:10.319567    5066 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee" Netns:"/var/run/netns/da84d06e-c29e-4fd3-bc61-4ace94ffe2ea" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get "https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s": context deadline exceeded
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]:  > pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:10.319925 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:10.319657    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress(eb333381-0634-4abe-93b2-121bc89433b8)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd_openshift-ingress_eb333381-0634-4abe-93b2-121bc89433b8_0(58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee): error adding pod openshift-ingress_e2e-mt-71335882-openshift-default-f77dbd9c-t22qd to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee\\\" Netns:\\\"/var/run/netns/da84d06e-c29e-4fd3-bc61-4ace94ffe2ea\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-mt-71335882-openshift-default-f77dbd9c-t22qd;K8S_POD_INFRA_CONTAINER_ID=58e2843c8f59557d87df572fa110b3d1e9d2d39303c33efac1317529ec05faee;K8S_POD_UID=eb333381-0634-4abe-93b2-121bc89433b8\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd] networking: Multus: [openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd/eb333381-0634-4abe-93b2-121bc89433b8]: error waiting for pod: Get \\\"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8"
Nov 01 20:32:13.041346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.041309    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:32:13.041909 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.041896    5066 cpu_manager.go:414] "containerMap: removing container" podUID="15ba9911-cef5-40aa-9351-203bfe11a828" containerName="main"
Nov 01 20:32:13.041953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.041911    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ba9911-cef5-40aa-9351-203bfe11a828" containerName="main"
Nov 01 20:32:13.042059 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.042047    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ba9911-cef5-40aa-9351-203bfe11a828" containerName="main"
Nov 01 20:32:13.045803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.045787    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.100726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.100694    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-393dc263-openshift-default-dockercfg-ssfv2\""
Nov 01 20:32:13.140013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.139979    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140032    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140230    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140281    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140367    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140415    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjvsc\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140453    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140697    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.140972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.140952    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.156401 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.156371    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:32:13.242878 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.242844    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.242910    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.242988    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243038    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjvsc\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243076    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243150    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243191    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243227    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243237    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243261    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243357 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243332    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243585 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243533    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243655 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243633    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.243980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.243951    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.245552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.245527    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.245678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.245660    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.299799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.299730    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.300374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.300347    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjvsc\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-zzdm9\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.360241 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.360208    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:13.797429 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.797404    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:32:13.798068 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:32:13.798046    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcefa44b_f625_4c21_a3b1_2a1a2803ec4d.slice/crio-3e072afa5ba3e113251658ccc8ac62e2bdff1a15f545986c7bbd6ecc129b4224 WatchSource:0}: Error finding container 3e072afa5ba3e113251658ccc8ac62e2bdff1a15f545986c7bbd6ecc129b4224: Status 404 returned error can't find the container with id 3e072afa5ba3e113251658ccc8ac62e2bdff1a15f545986c7bbd6ecc129b4224
Nov 01 20:32:13.800375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.800345    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:13.800460 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.800406    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:13.800460 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:13.800433    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:14.188080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.187998    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" event={"ID":"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d","Type":"ContainerStarted","Data":"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"}
Nov 01 20:32:14.188080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.188057    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" event={"ID":"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d","Type":"ContainerStarted","Data":"3e072afa5ba3e113251658ccc8ac62e2bdff1a15f545986c7bbd6ecc129b4224"}
Nov 01 20:32:14.312364 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.312277    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podStartSLOduration=2.312259443 podStartE2EDuration="2.312259443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:14.304818228 +0000 UTC m=+1787.689715819" watchObservedRunningTime="2026-11-01 20:32:14.312259443 +0000 UTC m=+1787.697157032"
Nov 01 20:32:14.363010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.362975    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:14.365085 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.365054    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:14.365653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:14.365326    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:15.386532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:15.360857    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:15.386532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:15.360912    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:16.396210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:16.396173    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:16.396804 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:16.396769    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:17.383518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:17.373849    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:17.383518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:17.373929    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:18.373540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:18.364825    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:18.373540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:18.364899    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:19.360786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:19.360751    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 20:32:19.361076 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:19.361044    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 20:32:20.366772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:20.366730    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:20.367794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:20.367770    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:20.368978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:20.368959    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:32:21.629864 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:21.629826    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:32:22.723209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:22.723172    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:22.723703 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:22.723602    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:23.179360 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:32:23.163668    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb333381_0634_4abe_93b2_121bc89433b8.slice/crio-6f1d711c550a83355609d4d13886776a6ba87476bb1a9355ed96d24c2a673ff0 WatchSource:0}: Error finding container 6f1d711c550a83355609d4d13886776a6ba87476bb1a9355ed96d24c2a673ff0: Status 404 returned error can't find the container with id 6f1d711c550a83355609d4d13886776a6ba87476bb1a9355ed96d24c2a673ff0
Nov 01 20:32:23.179360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.165739    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:23.179360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.165804    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:23.179360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.165835    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:23.179360 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.165917    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:32:23.263482 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.263446    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" event={"ID":"eb333381-0634-4abe-93b2-121bc89433b8","Type":"ContainerStarted","Data":"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"}
Nov 01 20:32:23.263635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.263551    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" event={"ID":"eb333381-0634-4abe-93b2-121bc89433b8","Type":"ContainerStarted","Data":"6f1d711c550a83355609d4d13886776a6ba87476bb1a9355ed96d24c2a673ff0"}
Nov 01 20:32:23.383665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.383601    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podStartSLOduration=50.383581074 podStartE2EDuration="50.383581074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:23.379978343 +0000 UTC m=+1796.764875911" watchObservedRunningTime="2026-11-01 20:32:23.383581074 +0000 UTC m=+1796.768478665"
Nov 01 20:32:23.501593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:23.501554    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:24.501792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:24.501754    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:32:24.502190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:24.501815    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 20:32:25.501630 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:25.501592    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:32:25.501933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:25.501895    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 20:32:26.500634 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:26.500598    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:32:26.500831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:26.500672    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 20:32:27.051578 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:27.051192    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:32:27.126342 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:27.126313    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:32:27.501880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:27.501078    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:32:27.501880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:27.501135    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 20:32:28.198652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.198586    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:32:28.205452 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.205424    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.251182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.251146    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.278694    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.279763    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280331    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280422    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280477    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280527    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vff\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280605    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280701    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.282932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.280740    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.308639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.297167    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.317058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.317027    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:32:28.327797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.327768    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382250    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382318    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382361    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382395    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382443    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vff\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382485    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382555    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382600    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382636    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382700    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382735    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382789    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382853    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382884    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.382992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.382934    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.383558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.383001    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.383558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.383040    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.383558 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.383070    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7t7p\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.394561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.394322    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.394943 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.394868    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.396073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.396014    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.396480 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.396438    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.396993 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.396964    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.399311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.399290    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.400151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.400135    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.450796 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.450728    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.456950 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.456920    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vff\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-55w6z\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.502856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.502825    5066 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 20:32:28.503068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.503046    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 20:32:28.503638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.503609    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.503832 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.503810    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.503914 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.503853    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504002 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.503988    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504060 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504034    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504108    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504192 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504142    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7t7p\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504245 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504198    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504245 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504226    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.504576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.504458    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.505328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.505303    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.505568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.505548    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.506158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.506115    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.506352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.506331    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.510552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.509072    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.511740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.511712    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.556691 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.556666    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:28.606248 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.606194    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.608379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.608354    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7t7p\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-rqg7z\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:28.648782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:28.648742    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:29.032648 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.032616    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:32:29.033740 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:32:29.033714    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc1ef5f_0b11_4ea9_8c88_69308adbaba2.slice/crio-3e3f1b9ce7ecb00924b757f2308dcf18b8ab98be9e65fdd2aec2fb7d8d72f45e WatchSource:0}: Error finding container 3e3f1b9ce7ecb00924b757f2308dcf18b8ab98be9e65fdd2aec2fb7d8d72f45e: Status 404 returned error can't find the container with id 3e3f1b9ce7ecb00924b757f2308dcf18b8ab98be9e65fdd2aec2fb7d8d72f45e
Nov 01 20:32:29.036188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.036155    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.036298 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.036217    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.036298 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.036248    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.106058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.106028    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:32:29.108334 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:32:29.108306    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549ef32a_da22_4325_a7b0_4b3d178ee7ba.slice/crio-db5beb8a344dfda8bf0b5a39fa609a9c6896904672cd46ca7eb85ec135d07ff3 WatchSource:0}: Error finding container db5beb8a344dfda8bf0b5a39fa609a9c6896904672cd46ca7eb85ec135d07ff3: Status 404 returned error can't find the container with id db5beb8a344dfda8bf0b5a39fa609a9c6896904672cd46ca7eb85ec135d07ff3
Nov 01 20:32:29.111207 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.111159    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.111293 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.111253    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.111331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.111290    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:32:29.354169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.354096    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" event={"ID":"549ef32a-da22-4325-a7b0-4b3d178ee7ba","Type":"ContainerStarted","Data":"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"}
Nov 01 20:32:29.354597 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.354575    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" event={"ID":"549ef32a-da22-4325-a7b0-4b3d178ee7ba","Type":"ContainerStarted","Data":"db5beb8a344dfda8bf0b5a39fa609a9c6896904672cd46ca7eb85ec135d07ff3"}
Nov 01 20:32:29.356038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.356010    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" event={"ID":"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2","Type":"ContainerStarted","Data":"2853b26216172913030f4c4a2f41479769b9f7943df8a280c43bcb7cd64f5f2a"}
Nov 01 20:32:29.356154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.356057    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" event={"ID":"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2","Type":"ContainerStarted","Data":"3e3f1b9ce7ecb00924b757f2308dcf18b8ab98be9e65fdd2aec2fb7d8d72f45e"}
Nov 01 20:32:29.476458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.476404    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podStartSLOduration=1.47639055 podStartE2EDuration="1.47639055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:29.475656946 +0000 UTC m=+1802.860554547" watchObservedRunningTime="2026-11-01 20:32:29.47639055 +0000 UTC m=+1802.861288140"
Nov 01 20:32:29.507629 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.507596    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:29.508013 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.507990    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:29.509462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.509438    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:32:29.557656 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.557625    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:29.614740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.614642    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podStartSLOduration=1.614627579 podStartE2EDuration="1.614627579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:29.612064947 +0000 UTC m=+1802.996962539" watchObservedRunningTime="2026-11-01 20:32:29.614627579 +0000 UTC m=+1802.999525169"
Nov 01 20:32:29.650281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:29.650254    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:30.558610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:30.558575    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 20:32:30.559034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:30.558630    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 20:32:30.651010 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:30.650971    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 01 20:32:30.651159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:30.651025    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 01 20:32:31.557111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:31.557075    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 20:32:31.557287 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:31.557123    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 20:32:31.649728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:31.649695    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 01 20:32:31.650108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:31.649745    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 01 20:32:32.557380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:32.557343    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 20:32:32.557576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:32.557400    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 20:32:32.650179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:32.650141    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 01 20:32:32.650586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:32.650189    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 01 20:32:33.557943 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:33.557912    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 20:32:33.558195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:33.558174    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 20:32:33.650422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:33.650385    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 01 20:32:33.650821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:33.650439    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 01 20:32:34.557088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:34.557048    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 20:32:34.557266 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:34.557103    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 20:32:34.650383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:34.650347    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Nov 01 20:32:34.650572 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:34.650399    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Nov 01 20:32:35.561987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.561956    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:35.562395 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.562364    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:35.563227 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.563204    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:32:35.654461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.654436    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:35.654849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.654761    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:35.655456 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:32:35.655441    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:32:51.627786 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:32:51.627750    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:33:03.603092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:03.603055    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:33:08.603542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.603386    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" podUID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" containerName="payload-processing" containerID="cri-o://a01be44721946c78057450816ae2a2191ebc573b0b9236a3dd2569a1ffe07194" gracePeriod=25
Nov 01 20:33:08.751104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.751070    5066 generic.go:356] "Generic (PLEG): container finished" podID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" containerID="a01be44721946c78057450816ae2a2191ebc573b0b9236a3dd2569a1ffe07194" exitCode=0
Nov 01 20:33:08.751264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.751127    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" event={"ID":"7f8b9863-69b2-4ff6-a302-05b1e094d22f","Type":"ContainerDied","Data":"a01be44721946c78057450816ae2a2191ebc573b0b9236a3dd2569a1ffe07194"}
Nov 01 20:33:08.878135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.878108    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:33:08.932863 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.932836    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\") pod \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") "
Nov 01 20:33:08.933026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.932882    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\") pod \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") "
Nov 01 20:33:08.933026 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933006    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\") pod \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") "
Nov 01 20:33:08.933103 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933088    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\") pod \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\" (UID: \"7f8b9863-69b2-4ff6-a302-05b1e094d22f\") "
Nov 01 20:33:08.933256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933229    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume" pod "7f8b9863-69b2-4ff6-a302-05b1e094d22f" (UID: "7f8b9863-69b2-4ff6-a302-05b1e094d22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:08.933305 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933242    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume" pod "7f8b9863-69b2-4ff6-a302-05b1e094d22f" (UID: "7f8b9863-69b2-4ff6-a302-05b1e094d22f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:08.933561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933537    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:08.933695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.933563    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f8b9863-69b2-4ff6-a302-05b1e094d22f-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:08.935193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.935168    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials" pod "7f8b9863-69b2-4ff6-a302-05b1e094d22f" (UID: "7f8b9863-69b2-4ff6-a302-05b1e094d22f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:08.935298 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:08.935203    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp" pod "7f8b9863-69b2-4ff6-a302-05b1e094d22f" (UID: "7f8b9863-69b2-4ff6-a302-05b1e094d22f"). InnerVolumeSpecName "kube-api-access-474qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:09.034565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.034532    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.034565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.034560    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-474qp\" (UniqueName: \"kubernetes.io/projected/7f8b9863-69b2-4ff6-a302-05b1e094d22f-kube-api-access-474qp\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:09.759082 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.759051    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"
Nov 01 20:33:09.759513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.759086    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz" event={"ID":"7f8b9863-69b2-4ff6-a302-05b1e094d22f","Type":"ContainerDied","Data":"1ab7894da2cd2478c0f6dad028580b08ddea5ada0054d8fbebd9cdaa36db628f"}
Nov 01 20:33:09.759513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.759147    5066 scope.go:122] "RemoveContainer" containerID="a01be44721946c78057450816ae2a2191ebc573b0b9236a3dd2569a1ffe07194"
Nov 01 20:33:09.922898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.922868    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:33:09.971600 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:09.971564    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586b9ccgz"]
Nov 01 20:33:10.731802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:10.731763    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" path="/var/lib/kubelet/pods/7f8b9863-69b2-4ff6-a302-05b1e094d22f/volumes"
Nov 01 20:33:16.660987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.660955    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:33:16.661724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.661705    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" containerName="payload-processing"
Nov 01 20:33:16.661786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.661725    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" containerName="payload-processing"
Nov 01 20:33:16.661869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.661860    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8b9863-69b2-4ff6-a302-05b1e094d22f" containerName="payload-processing"
Nov 01 20:33:16.666836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.666820    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.721524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.721487    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:33:16.760523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.760472    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.760704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.760674    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.760869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.760851    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.760930 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.760893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7zr\" (UniqueName: \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.766671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.766646    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f154f7-serving-cert\""
Nov 01 20:33:16.863022 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.862973    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.863261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.863205    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.863395 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.863370    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.863671 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.863647    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7zr\" (UniqueName: \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.863836 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:16.863535    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-serving-cert: secret "maas-api-e2e-explicit-f154f7-serving-cert" not found
Nov 01 20:33:16.864149 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:16.864126    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls podName:b00046a0-80ff-41aa-b9fc-a9cca32c0b01 nodeName:}" failed. No retries permitted until 2026-11-01 20:33:17.364102773 +0000 UTC m=+1850.749000346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls") pod "maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" (UID: "b00046a0-80ff-41aa-b9fc-a9cca32c0b01") : secret "maas-api-e2e-explicit-f154f7-serving-cert" not found
Nov 01 20:33:16.864787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.864762    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.866424 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.866384    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:16.930907 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:16.930798    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7zr\" (UniqueName: \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:17.372046 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:17.372006    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:17.374996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:17.374966    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") pod \"maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:17.580744 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:17.580714    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:17.957815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:17.957784    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:33:17.959181 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:33:17.959148    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00046a0_80ff_41aa_b9fc_a9cca32c0b01.slice/crio-a381d94a913d8af0ceb9c86895ff399f41528ef7b3029c55e5a99030fb834390 WatchSource:0}: Error finding container a381d94a913d8af0ceb9c86895ff399f41528ef7b3029c55e5a99030fb834390: Status 404 returned error can't find the container with id a381d94a913d8af0ceb9c86895ff399f41528ef7b3029c55e5a99030fb834390
Nov 01 20:33:18.849810 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:18.849773    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" event={"ID":"b00046a0-80ff-41aa-b9fc-a9cca32c0b01","Type":"ContainerStarted","Data":"a381d94a913d8af0ceb9c86895ff399f41528ef7b3029c55e5a99030fb834390"}
Nov 01 20:33:19.861748 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:19.861711    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" event={"ID":"b00046a0-80ff-41aa-b9fc-a9cca32c0b01","Type":"ContainerStarted","Data":"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"}
Nov 01 20:33:19.862281 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:19.862257    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:19.977781 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:19.977727    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" podStartSLOduration=3.002115531 podStartE2EDuration="3.977713072s" totalImagesPullingTime="975.597541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:19.97585607 +0000 UTC m=+1853.360753661" watchObservedRunningTime="2026-11-01 20:33:19.977713072 +0000 UTC m=+1853.362610661"
Nov 01 20:33:21.630260 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:21.630220    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:33:25.874353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:25.874321    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:33:47.600184 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.600151    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:33:47.600811 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.600383    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" containerID="cri-o://9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3" gracePeriod=30
Nov 01 20:33:47.735595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.735560    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:33:47.736119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.736079    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" containerID="cri-o://ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7" gracePeriod=30
Nov 01 20:33:47.755844 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.754719    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:33:47.755844 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:47.754965    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" containerID="cri-o://2853b26216172913030f4c4a2f41479769b9f7943df8a280c43bcb7cd64f5f2a" gracePeriod=30
Nov 01 20:33:48.289212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:48.289165    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:33:48.289710 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:48.289672    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy" containerID="cri-o://1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625" gracePeriod=30
Nov 01 20:33:48.302937 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:48.302904    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:33:48.303154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:48.303116    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy" containerID="cri-o://1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4" gracePeriod=30
Nov 01 20:33:49.516126 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.516080    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:33:49.520385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.520361    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.553685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.553649    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glg4\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.553881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.553800    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.553938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.553879    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.553938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.553906    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.554037 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.553984    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.554093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.554062    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.554141 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.554096    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.554217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.554197    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.554269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.554229    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.587237 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.587202    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:33:49.654949 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.654918    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.654963    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655010    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655025    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655050    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glg4\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655155    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655209    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655244    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655297    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655584 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655402    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655644 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655603    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655775    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655902 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655841    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.655986 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.655959    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.657695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.657675    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.658092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.658073    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.711555 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.711521    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glg4\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.713862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.713830    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:49.837564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:49.837462    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:50.270873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.270837    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:33:50.310800 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:33:50.310769    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf355f0_37ee_48f4_83ce_930dde15737a.slice/crio-5b9c3d07f2f39efda920f511b2836e3fe81a28355c92b7d4f17189ed3ce3a56c WatchSource:0}: Error finding container 5b9c3d07f2f39efda920f511b2836e3fe81a28355c92b7d4f17189ed3ce3a56c: Status 404 returned error can't find the container with id 5b9c3d07f2f39efda920f511b2836e3fe81a28355c92b7d4f17189ed3ce3a56c
Nov 01 20:33:50.313311 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.313279    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:33:50.313402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.313360    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:33:50.313480 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.313404    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:33:50.370133 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.370103    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:50.370296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.370154    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:50.565014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.564929    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-55w6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:50.565014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.564991    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:50.657251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.657213    5066 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:50.657414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:50.657292    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:51.102030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.101985    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" event={"ID":"baf355f0-37ee-48f4-83ce-930dde15737a","Type":"ContainerStarted","Data":"f2c18e6f59d8a79afcaae30947b128b8d21a14e22a3fda388a6e53c162da1260"}
Nov 01 20:33:51.102247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.102049    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" event={"ID":"baf355f0-37ee-48f4-83ce-930dde15737a","Type":"ContainerStarted","Data":"5b9c3d07f2f39efda920f511b2836e3fe81a28355c92b7d4f17189ed3ce3a56c"}
Nov 01 20:33:51.219949 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.219895    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podStartSLOduration=2.219876984 podStartE2EDuration="2.219876984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:51.217146016 +0000 UTC m=+1884.602043608" watchObservedRunningTime="2026-11-01 20:33:51.219876984 +0000 UTC m=+1884.604774574"
Nov 01 20:33:51.633171 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:51.633137    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:33:51.759234 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.759199    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:33:51.759412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.759387    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" containerID="cri-o://91c349a19556a9d3ca46eab28f8fe855c8196fd25c3347493b16db5a1f3d53e0" gracePeriod=30
Nov 01 20:33:51.808913 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.808882    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:33:51.809113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.809078    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" containerID="cri-o://0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b" gracePeriod=30
Nov 01 20:33:51.838200 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.838170    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:51.839695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.839668    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 20:33:51.839814 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:51.839714    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 20:33:52.838217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:52.838184    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 20:33:52.838759 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:52.838238    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 20:33:52.999676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:52.985620    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:33:53.023218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.022952    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.023425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.023376    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.023515 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.023458    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.023579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.023513    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.023811 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.023788    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.024468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.024377    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.025409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.025045    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.025900 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.025853    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.028878 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.026540    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:53.028878 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.027558    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.028878 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.028765    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.030302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.029073    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.030302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.029115    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.030302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.029234    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\") pod \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\" (UID: \"549ef32a-da22-4325-a7b0-4b3d178ee7ba\") "
Nov 01 20:33:53.030302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.030268    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:53.030873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.030843    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031375    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031398    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031411    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031424    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031439    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031453    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.031508 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.031466    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.032579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.032536    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.032699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.032669    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p" pod "549ef32a-da22-4325-a7b0-4b3d178ee7ba" (UID: "549ef32a-da22-4325-a7b0-4b3d178ee7ba"). InnerVolumeSpecName "kube-api-access-r7t7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.123556 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.123525    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:33:53.127032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.126999    5066 generic.go:356] "Generic (PLEG): container finished" podID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerID="9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3" exitCode=0
Nov 01 20:33:53.127193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.127094    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" event={"ID":"549ef32a-da22-4325-a7b0-4b3d178ee7ba","Type":"ContainerDied","Data":"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"}
Nov 01 20:33:53.127193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.127145    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z" event={"ID":"549ef32a-da22-4325-a7b0-4b3d178ee7ba","Type":"ContainerDied","Data":"db5beb8a344dfda8bf0b5a39fa609a9c6896904672cd46ca7eb85ec135d07ff3"}
Nov 01 20:33:53.127193 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.127172    5066 scope.go:122] "RemoveContainer" containerID="9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"
Nov 01 20:33:53.127352 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.127341    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"
Nov 01 20:33:53.133375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.133326    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549ef32a-da22-4325-a7b0-4b3d178ee7ba-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.133375 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.133352    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7t7p\" (UniqueName: \"kubernetes.io/projected/549ef32a-da22-4325-a7b0-4b3d178ee7ba-kube-api-access-r7t7p\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.134574 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.134513    5066 generic.go:356] "Generic (PLEG): container finished" podID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerID="2853b26216172913030f4c4a2f41479769b9f7943df8a280c43bcb7cd64f5f2a" exitCode=0
Nov 01 20:33:53.134789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.134737    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" event={"ID":"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2","Type":"ContainerDied","Data":"2853b26216172913030f4c4a2f41479769b9f7943df8a280c43bcb7cd64f5f2a"}
Nov 01 20:33:53.136685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.136658    5066 generic.go:356] "Generic (PLEG): container finished" podID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerID="ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7" exitCode=0
Nov 01 20:33:53.137117 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.137093    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"
Nov 01 20:33:53.137265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.137243    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" event={"ID":"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d","Type":"ContainerDied","Data":"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"}
Nov 01 20:33:53.137336 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.137291    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9" event={"ID":"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d","Type":"ContainerDied","Data":"3e072afa5ba3e113251658ccc8ac62e2bdff1a15f545986c7bbd6ecc129b4224"}
Nov 01 20:33:53.153922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.153098    5066 scope.go:122] "RemoveContainer" containerID="9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"
Nov 01 20:33:53.153922 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:53.153593    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3\": container with ID starting with 9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3 not found: ID does not exist" containerID="9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"
Nov 01 20:33:53.153922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.153634    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3"} err="failed to get container status \"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3\": rpc error: code = NotFound desc = could not find container \"9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3\": container with ID starting with 9ca6d3dbc7de24315eedfe1eea661b72919f7740071c072d3da7ffa0cf9817a3 not found: ID does not exist"
Nov 01 20:33:53.153922 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.153659    5066 scope.go:122] "RemoveContainer" containerID="ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"
Nov 01 20:33:53.218077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.218032    5066 scope.go:122] "RemoveContainer" containerID="ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"
Nov 01 20:33:53.218655 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:53.218576    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7\": container with ID starting with ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7 not found: ID does not exist" containerID="ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"
Nov 01 20:33:53.218655 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.218615    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7"} err="failed to get container status \"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7\": rpc error: code = NotFound desc = could not find container \"ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7\": container with ID starting with ec63782096da1baacc30e404719934010b177ef956fbfc415df1a48e4c4e21f7 not found: ID does not exist"
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234059    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234151    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234203    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234266    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234435    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234517    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234571    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234620    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.235455 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.234651    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\") pod \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\" (UID: \"dcefa44b-f625-4c21-a3b1-2a1a2803ec4d\") "
Nov 01 20:33:53.236027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.235928    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.236377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.236159    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.236831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.236556    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.239425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.239002    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "kube-api-access-gjvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.239425 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.239208    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.240348 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.240322    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:53.242549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.240903    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.243582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.243560    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:33:53.244672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.244642    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.245023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.244998    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo" pod "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" (UID: "dcefa44b-f625-4c21-a3b1-2a1a2803ec4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341400    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341533    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341622    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341684    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341721    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341788    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341849    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341903    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.341935    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\") pod \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\" (UID: \"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2\") "
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342436    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342457    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjvsc\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-kube-api-access-gjvsc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342472    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342486    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342539    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342553    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342568    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342584    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.342600    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.395836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.377839    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.413557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.412643    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.413557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.412931    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.431562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.416726    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:53.431562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.431206    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.435102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.435053    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.435829 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.435720    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "kube-api-access-29vff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.436457 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.436377    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:53.436795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.436762    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy" pod "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" (UID: "0cc1ef5f-0b11-4ea9-8c88-69308adbaba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.438031 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.437976    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445008    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vff\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-kube-api-access-29vff\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445040    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445056    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445070    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445083    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445096    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445112    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445125    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.446421 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.445137    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.451568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.451529    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-rqg7z"]
Nov 01 20:33:53.503852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.503578    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:53.503852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.503643    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:53.530993 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.530951    5066 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:53.531150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.531027    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:53.701596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.688911    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:33:53.706898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.706836    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-zzdm9"]
Nov 01 20:33:53.797780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.797744    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:33:53.837960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.837925    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 20:33:53.838194 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.838163    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 20:33:53.850978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.850947    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:33:53.856217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856194    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856223    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856260    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856285    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856331    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856359    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856380    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856395    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856416    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856446    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856465    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856514    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856562    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856621    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856647    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856673    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856696    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\") pod \"eb333381-0634-4abe-93b2-121bc89433b8\" (UID: \"eb333381-0634-4abe-93b2-121bc89433b8\") "
Nov 01 20:33:53.856756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856750    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\") pod \"77ea3fef-03db-4c45-9a05-5cea65b76618\" (UID: \"77ea3fef-03db-4c45-9a05-5cea65b76618\") "
Nov 01 20:33:53.857158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.856775    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.857158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.857131    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.857240 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.857226    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.857410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.857286    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.857464 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.857449    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.857661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.857600    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.858236 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.858212    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:53.858678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.858655    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.859214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.859194    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:53.859377 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.859358    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.859579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.859546    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.859832 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.859812    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.859933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.859891    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.860098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.860072    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "kube-api-access-bjg87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.875695 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.869022    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:53.895215 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.894628    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:53.895564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.895523    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:53.896586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.896543    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token" pod "77ea3fef-03db-4c45-9a05-5cea65b76618" (UID: "77ea3fef-03db-4c45-9a05-5cea65b76618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.897215 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.897177    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx" pod "eb333381-0634-4abe-93b2-121bc89433b8" (UID: "eb333381-0634-4abe-93b2-121bc89433b8"). InnerVolumeSpecName "kube-api-access-th7zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:53.958591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958483    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958539    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7zx\" (UniqueName: \"kubernetes.io/projected/eb333381-0634-4abe-93b2-121bc89433b8-kube-api-access-th7zx\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958557    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958572    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958586    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958598    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb333381-0634-4abe-93b2-121bc89433b8-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958610    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958622    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb333381-0634-4abe-93b2-121bc89433b8-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958635    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958648    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjg87\" (UniqueName: \"kubernetes.io/projected/77ea3fef-03db-4c45-9a05-5cea65b76618-kube-api-access-bjg87\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958662    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958675    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958690    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958704    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ea3fef-03db-4c45-9a05-5cea65b76618-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958718    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958731    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb333381-0634-4abe-93b2-121bc89433b8-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:53.958977 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:53.958778    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ea3fef-03db-4c45-9a05-5cea65b76618-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:54.146409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.146374    5066 generic.go:356] "Generic (PLEG): container finished" podID="eb333381-0634-4abe-93b2-121bc89433b8" containerID="1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4" exitCode=0
Nov 01 20:33:54.146593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.146419    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"
Nov 01 20:33:54.146593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.146439    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" event={"ID":"eb333381-0634-4abe-93b2-121bc89433b8","Type":"ContainerDied","Data":"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"}
Nov 01 20:33:54.146593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.146478    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd" event={"ID":"eb333381-0634-4abe-93b2-121bc89433b8","Type":"ContainerDied","Data":"6f1d711c550a83355609d4d13886776a6ba87476bb1a9355ed96d24c2a673ff0"}
Nov 01 20:33:54.146593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.146522    5066 scope.go:122] "RemoveContainer" containerID="1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"
Nov 01 20:33:54.148948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.148927    5066 generic.go:356] "Generic (PLEG): container finished" podID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerID="1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625" exitCode=0
Nov 01 20:33:54.149056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.148991    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" event={"ID":"77ea3fef-03db-4c45-9a05-5cea65b76618","Type":"ContainerDied","Data":"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"}
Nov 01 20:33:54.149056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.149019    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9" event={"ID":"77ea3fef-03db-4c45-9a05-5cea65b76618","Type":"ContainerDied","Data":"d7594900041cc962c26adb88018ae5cf368e12d8ff563775d4c24ff4ec9ffb05"}
Nov 01 20:33:54.149181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.149064    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"
Nov 01 20:33:54.155047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.155002    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"
Nov 01 20:33:54.155047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.155026    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z" event={"ID":"0cc1ef5f-0b11-4ea9-8c88-69308adbaba2","Type":"ContainerDied","Data":"3e3f1b9ce7ecb00924b757f2308dcf18b8ab98be9e65fdd2aec2fb7d8d72f45e"}
Nov 01 20:33:54.173545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.160730    5066 scope.go:122] "RemoveContainer" containerID="1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"
Nov 01 20:33:54.174104 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:54.173897    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4\": container with ID starting with 1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4 not found: ID does not exist" containerID="1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"
Nov 01 20:33:54.174104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.173943    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4"} err="failed to get container status \"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4\": rpc error: code = NotFound desc = could not find container \"1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4\": container with ID starting with 1996d4652e120c5b97ca33f57699ced636cdaa022a2f7f5b1f987663b9e59ee4 not found: ID does not exist"
Nov 01 20:33:54.174104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.173973    5066 scope.go:122] "RemoveContainer" containerID="1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"
Nov 01 20:33:54.202416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.202393    5066 scope.go:122] "RemoveContainer" containerID="1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"
Nov 01 20:33:54.202763 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:54.202743    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625\": container with ID starting with 1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625 not found: ID does not exist" containerID="1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"
Nov 01 20:33:54.202842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.202774    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625"} err="failed to get container status \"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625\": rpc error: code = NotFound desc = could not find container \"1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625\": container with ID starting with 1b1b67329e6e750a13d86ab7eab91691f041c75ce54ded6354b2fe55b6b10625 not found: ID does not exist"
Nov 01 20:33:54.202842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.202796    5066 scope.go:122] "RemoveContainer" containerID="2853b26216172913030f4c4a2f41479769b9f7943df8a280c43bcb7cd64f5f2a"
Nov 01 20:33:54.316316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.316282    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:33:54.319552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.319523    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-br4r9"]
Nov 01 20:33:54.435522 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.435458    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:33:54.442980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.442941    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-t22qd"]
Nov 01 20:33:54.566519 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.560547    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:33:54.566811 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.566788    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-55w6z"]
Nov 01 20:33:54.741122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.741079    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" path="/var/lib/kubelet/pods/0cc1ef5f-0b11-4ea9-8c88-69308adbaba2/volumes"
Nov 01 20:33:54.741669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.741648    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" path="/var/lib/kubelet/pods/549ef32a-da22-4325-a7b0-4b3d178ee7ba/volumes"
Nov 01 20:33:54.742128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.742110    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" path="/var/lib/kubelet/pods/77ea3fef-03db-4c45-9a05-5cea65b76618/volumes"
Nov 01 20:33:54.742926 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.742906    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" path="/var/lib/kubelet/pods/dcefa44b-f625-4c21-a3b1-2a1a2803ec4d/volumes"
Nov 01 20:33:54.743366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.743349    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb333381-0634-4abe-93b2-121bc89433b8" path="/var/lib/kubelet/pods/eb333381-0634-4abe-93b2-121bc89433b8/volumes"
Nov 01 20:33:54.837905 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.837827    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 20:33:54.838104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:54.838081    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 20:33:55.838892 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:55.838847    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 01 20:33:55.839268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:55.838909    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 01 20:33:57.079475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.077861    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:57.079475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.079259    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:57.081300 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.081246    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:33:57.132057 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.132032    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:33:57.219079 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.219046    5066 generic.go:356] "Generic (PLEG): container finished" podID="115a3953-6c37-497b-888b-489b2e8effe9" containerID="91c349a19556a9d3ca46eab28f8fe855c8196fd25c3347493b16db5a1f3d53e0" exitCode=0
Nov 01 20:33:57.219267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.219117    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" event={"ID":"115a3953-6c37-497b-888b-489b2e8effe9","Type":"ContainerDied","Data":"91c349a19556a9d3ca46eab28f8fe855c8196fd25c3347493b16db5a1f3d53e0"}
Nov 01 20:33:57.220532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.220488    5066 generic.go:356] "Generic (PLEG): container finished" podID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerID="0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b" exitCode=0
Nov 01 20:33:57.220658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.220561    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"
Nov 01 20:33:57.220658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.220604    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" event={"ID":"677006c1-b202-4a2e-bff8-cb6e7a5e70b2","Type":"ContainerDied","Data":"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"}
Nov 01 20:33:57.220658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.220655    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz" event={"ID":"677006c1-b202-4a2e-bff8-cb6e7a5e70b2","Type":"ContainerDied","Data":"e9952c2aa9f1e8829ff51e16ac28e97b1677d6dda9c5cfaeca4a0788dbd8315d"}
Nov 01 20:33:57.220824 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.220684    5066 scope.go:122] "RemoveContainer" containerID="0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"
Nov 01 20:33:57.227062 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227042    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227080    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227114    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227134    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227163    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227186    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227243    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227290    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.227371 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227324    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\") pod \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\" (UID: \"677006c1-b202-4a2e-bff8-cb6e7a5e70b2\") "
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227386    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227445    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227473    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227592    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227799    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.227962    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.228000    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.228016    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.228030    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.229106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.228044    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.230129 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.230102    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "kube-api-access-c5xpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:57.230129 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.230119    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:57.230312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.230294    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:57.230591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.230571    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy" pod "677006c1-b202-4a2e-bff8-cb6e7a5e70b2" (UID: "677006c1-b202-4a2e-bff8-cb6e7a5e70b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.231703 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.231686    5066 scope.go:122] "RemoveContainer" containerID="0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"
Nov 01 20:33:57.231971 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:33:57.231951    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b\": container with ID starting with 0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b not found: ID does not exist" containerID="0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"
Nov 01 20:33:57.232040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.231989    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b"} err="failed to get container status \"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b\": rpc error: code = NotFound desc = could not find container \"0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b\": container with ID starting with 0397b04ab359a961616eff8933bc6c994d76005b16d13ae1b99ea6df20ba845b not found: ID does not exist"
Nov 01 20:33:57.328756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.328724    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xpj\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-kube-api-access-c5xpj\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.328756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.328756    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.328932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.328770    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.328932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.328782    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677006c1-b202-4a2e-bff8-cb6e7a5e70b2-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.453263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.453234    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:33:57.531820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.531793    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.531828    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.531912    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.531939    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532001    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532045    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532084    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.532254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532094    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532162    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532284    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\") pod \"115a3953-6c37-497b-888b-489b2e8effe9\" (UID: \"115a3953-6c37-497b-888b-489b2e8effe9\") "
Nov 01 20:33:57.532453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532383    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.532698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532654    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.532780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532754    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.532971    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.533353    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.533374    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115a3953-6c37-497b-888b-489b2e8effe9-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.533394    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.533408    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.533828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.533420    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.535187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.535163    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:57.535379 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.535358    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "kube-api-access-j4pvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:57.535479 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.535464    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:57.536333 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.536316    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token" pod "115a3953-6c37-497b-888b-489b2e8effe9" (UID: "115a3953-6c37-497b-888b-489b2e8effe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:57.635111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.635040    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pvz\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-kube-api-access-j4pvz\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.635111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.635070    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115a3953-6c37-497b-888b-489b2e8effe9-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.635111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.635080    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115a3953-6c37-497b-888b-489b2e8effe9-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.635111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.635088    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115a3953-6c37-497b-888b-489b2e8effe9-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:33:57.707084 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.707050    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:33:57.711032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:57.711008    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vq9vz"]
Nov 01 20:33:58.235048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.235019    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"
Nov 01 20:33:58.235048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.235038    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp" event={"ID":"115a3953-6c37-497b-888b-489b2e8effe9","Type":"ContainerDied","Data":"81011096ebfa1f9617a4aec0276b9e216cf74a6397d5b8a3b55aee375abb0788"}
Nov 01 20:33:58.235588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.235091    5066 scope.go:122] "RemoveContainer" containerID="91c349a19556a9d3ca46eab28f8fe855c8196fd25c3347493b16db5a1f3d53e0"
Nov 01 20:33:58.399272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.399231    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:33:58.447666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.447635    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-9zpdp"]
Nov 01 20:33:58.731210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.731177    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115a3953-6c37-497b-888b-489b2e8effe9" path="/var/lib/kubelet/pods/115a3953-6c37-497b-888b-489b2e8effe9/volumes"
Nov 01 20:33:58.731771 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:33:58.731748    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" path="/var/lib/kubelet/pods/677006c1-b202-4a2e-bff8-cb6e7a5e70b2/volumes"
Nov 01 20:34:01.853932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:01.853897    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:34:01.854353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:01.854100    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" containerID="cri-o://e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b" gracePeriod=30
Nov 01 20:34:06.787876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:06.787836    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nzqvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:34:06.788268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:06.787893    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:34:08.125642 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.125613    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:34:08.275445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275406    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275511    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275535    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275587    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275628    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275722    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275734    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:08.275875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275815    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.275875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275857    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.276077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275890    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\") pod \"154811b0-d204-4fa5-96b4-8f648e70292d\" (UID: \"154811b0-d204-4fa5-96b4-8f648e70292d\") "
Nov 01 20:34:08.276077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.275903    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:08.276077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276039    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:08.276320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276293    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:08.276525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276469    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:08.276621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276597    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.276678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276619    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.276678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276635    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.276678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.276649    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.277801 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.277766    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:08.278149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.278118    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "kube-api-access-nzb8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:08.278267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.278228    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:34:08.278873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.278854    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy" pod "154811b0-d204-4fa5-96b4-8f648e70292d" (UID: "154811b0-d204-4fa5-96b4-8f648e70292d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:08.317106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.317074    5066 generic.go:356] "Generic (PLEG): container finished" podID="154811b0-d204-4fa5-96b4-8f648e70292d" containerID="e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b" exitCode=0
Nov 01 20:34:08.317267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.317140    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"
Nov 01 20:34:08.317267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.317171    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerDied","Data":"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"}
Nov 01 20:34:08.317267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.317240    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb" event={"ID":"154811b0-d204-4fa5-96b4-8f648e70292d","Type":"ContainerDied","Data":"6179b4a4fc96d6ef1fab12328d18f7a736104862f374fbe320605b397bdff083"}
Nov 01 20:34:08.317413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.317274    5066 scope.go:122] "RemoveContainer" containerID="e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"
Nov 01 20:34:08.330740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.330712    5066 scope.go:122] "RemoveContainer" containerID="e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"
Nov 01 20:34:08.342672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.342587    5066 scope.go:122] "RemoveContainer" containerID="e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"
Nov 01 20:34:08.342940 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:34:08.342894    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b\": container with ID starting with e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b not found: ID does not exist" containerID="e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"
Nov 01 20:34:08.343018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.342936    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b"} err="failed to get container status \"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b\": rpc error: code = NotFound desc = could not find container \"e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b\": container with ID starting with e8b6e5ad0f50b38d770775c0e90ee924d3ffb7f28bce97d1905fba85737d593b not found: ID does not exist"
Nov 01 20:34:08.343018 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.342961    5066 scope.go:122] "RemoveContainer" containerID="e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"
Nov 01 20:34:08.343365 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:34:08.343342    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898\": container with ID starting with e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898 not found: ID does not exist" containerID="e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"
Nov 01 20:34:08.343441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.343374    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898"} err="failed to get container status \"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898\": rpc error: code = NotFound desc = could not find container \"e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898\": container with ID starting with e2b7f2582c96acc017c2bb7bc9ad38f473df9cbe8c3526df5e59bd184222e898 not found: ID does not exist"
Nov 01 20:34:08.377708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.377687    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154811b0-d204-4fa5-96b4-8f648e70292d-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.377708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.377710    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154811b0-d204-4fa5-96b4-8f648e70292d-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.377859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.377721    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.377859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.377730    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzb8m\" (UniqueName: \"kubernetes.io/projected/154811b0-d204-4fa5-96b4-8f648e70292d-kube-api-access-nzb8m\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.377859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.377740    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154811b0-d204-4fa5-96b4-8f648e70292d-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:08.482740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.482636    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:34:08.485615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.485589    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nzqvb"]
Nov 01 20:34:08.731751 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.731723    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" path="/var/lib/kubelet/pods/154811b0-d204-4fa5-96b4-8f648e70292d/volumes"
Nov 01 20:34:08.974888 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:08.974849    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:34:21.628927 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:34:21.628887    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:34:23.996278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:23.996233    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" containerName="main" containerID="cri-o://af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461" gracePeriod=45
Nov 01 20:34:24.018919 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.018885    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": dial tcp 10.132.0.108:8000: connect: connection refused"
Nov 01 20:34:24.267675 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.267651    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:34:24.369109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369076    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369133    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369194    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\" (UniqueName: \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369226    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369257    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369439 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369281    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\" (UniqueName: \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\") pod \"167f2990-e934-4d04-8132-83cc7b0b8843\" (UID: \"167f2990-e934-4d04-8132-83cc7b0b8843\") "
Nov 01 20:34:24.369439 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369387    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:24.369583 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369563    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:24.369644 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369584    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:24.369802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369776    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.369802 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369797    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.369942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.369812    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.371224 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.371199    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:24.371745 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.371718    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:34:24.371858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.371723    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9" pod "167f2990-e934-4d04-8132-83cc7b0b8843" (UID: "167f2990-e934-4d04-8132-83cc7b0b8843"). InnerVolumeSpecName "kube-api-access-9lhq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:24.428482 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.428450    5066 generic.go:356] "Generic (PLEG): container finished" podID="167f2990-e934-4d04-8132-83cc7b0b8843" containerID="af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461" exitCode=0
Nov 01 20:34:24.428669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.428530    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"
Nov 01 20:34:24.428669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.428546    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" event={"ID":"167f2990-e934-4d04-8132-83cc7b0b8843","Type":"ContainerDied","Data":"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"}
Nov 01 20:34:24.428669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.428608    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4" event={"ID":"167f2990-e934-4d04-8132-83cc7b0b8843","Type":"ContainerDied","Data":"c6ead1eed90305185f35be8df0bb57f06e79aa0c19ff96ba9f3b2479d76c7d44"}
Nov 01 20:34:24.428669 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.428636    5066 scope.go:122] "RemoveContainer" containerID="af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"
Nov 01 20:34:24.439936 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.439915    5066 scope.go:122] "RemoveContainer" containerID="af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"
Nov 01 20:34:24.440228 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:34:24.440209    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461\": container with ID starting with af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461 not found: ID does not exist" containerID="af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"
Nov 01 20:34:24.440283 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.440235    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461"} err="failed to get container status \"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461\": rpc error: code = NotFound desc = could not find container \"af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461\": container with ID starting with af6c0fdeb3ac8649736ca05b181146e569a0afe0742dcb46c27f2fa3c5b21461 not found: ID does not exist"
Nov 01 20:34:24.475711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.475652    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167f2990-e934-4d04-8132-83cc7b0b8843-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.475711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.475695    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167f2990-e934-4d04-8132-83cc7b0b8843-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.475711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.475713    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhq9\" (UniqueName: \"kubernetes.io/projected/167f2990-e934-4d04-8132-83cc7b0b8843-kube-api-access-9lhq9\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:34:24.597458 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.597355    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:34:24.601229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.601202    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-df41b8/sub-test-model-df41b8-kserve-699d9fd455-h9rg4"]
Nov 01 20:34:24.731681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:24.731648    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" path="/var/lib/kubelet/pods/167f2990-e934-4d04-8132-83cc7b0b8843/volumes"
Nov 01 20:34:32.114579 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.114548    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:34:32.115118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115104    5066 cpu_manager.go:414] "containerMap: removing container" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy"
Nov 01 20:34:32.115118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115118    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115129    5066 cpu_manager.go:414] "containerMap: removing container" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115135    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115146    5066 cpu_manager.go:414] "containerMap: removing container" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" containerName="main"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115151    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" containerName="main"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115161    5066 cpu_manager.go:414] "containerMap: removing container" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115165    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115173    5066 cpu_manager.go:414] "containerMap: removing container" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115177    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115193    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115198    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115205    5066 cpu_manager.go:414] "containerMap: removing container" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy"
Nov 01 20:34:32.115212 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115209    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115218    5066 cpu_manager.go:414] "containerMap: removing container" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115232    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115245    5066 cpu_manager.go:414] "containerMap: removing container" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115250    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115256    5066 cpu_manager.go:414] "containerMap: removing container" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115260    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115350    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc1ef5f-0b11-4ea9-8c88-69308adbaba2" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115357    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="677006c1-b202-4a2e-bff8-cb6e7a5e70b2" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115366    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcefa44b-f625-4c21-a3b1-2a1a2803ec4d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115374    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115381    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="154811b0-d204-4fa5-96b4-8f648e70292d" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115389    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="115a3953-6c37-497b-888b-489b2e8effe9" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115394    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ea3fef-03db-4c45-9a05-5cea65b76618" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115400    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="167f2990-e934-4d04-8132-83cc7b0b8843" containerName="main"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115406    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="549ef32a-da22-4325-a7b0-4b3d178ee7ba" containerName="istio-proxy"
Nov 01 20:34:32.115605 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.115412    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb333381-0634-4abe-93b2-121bc89433b8" containerName="istio-proxy"
Nov 01 20:34:32.120750 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.120728    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.173068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.173037    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 20:34:32.173891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.173867    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.173914    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174097 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.173968    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174109    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7nf\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174167    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174191    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174233    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174256    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.174422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.174355    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.227619 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.227567    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:34:32.275850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.275816    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276004 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.275895    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276066 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276044    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276121    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276180    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7nf\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276231    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276236    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276256    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276305    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276346 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276328    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276380    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276614    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276701    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.276884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.276858    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.278509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.278476    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.278590 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.278517    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.332797 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.332772    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7nf\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.332917 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.332774    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-z5gnd\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.433952 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.433875    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:32.877340 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.877311    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:34:32.878447 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:34:32.878422    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7c06d8_e6eb_4575_a7d7_72457c2cabfb.slice/crio-1670ba1475338cb9a9c4909f684c8d2eb736b14b5d5a260f38b2fb18b228794c WatchSource:0}: Error finding container 1670ba1475338cb9a9c4909f684c8d2eb736b14b5d5a260f38b2fb18b228794c: Status 404 returned error can't find the container with id 1670ba1475338cb9a9c4909f684c8d2eb736b14b5d5a260f38b2fb18b228794c
Nov 01 20:34:32.880889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.880859    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:34:32.880987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.880939    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:34:32.881051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:32.880999    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:34:33.494960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:33.494917    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" event={"ID":"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb","Type":"ContainerStarted","Data":"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"}
Nov 01 20:34:33.495443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:33.494977    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" event={"ID":"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb","Type":"ContainerStarted","Data":"1670ba1475338cb9a9c4909f684c8d2eb736b14b5d5a260f38b2fb18b228794c"}
Nov 01 20:34:33.612840 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:33.612774    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podStartSLOduration=1.612748388 podStartE2EDuration="1.612748388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:33.611594767 +0000 UTC m=+1926.996492363" watchObservedRunningTime="2026-11-01 20:34:33.612748388 +0000 UTC m=+1926.997645981"
Nov 01 20:34:34.434757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:34.434714    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:34.436682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:34.436654    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:34.436815 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:34.436713    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:35.434627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:35.434593    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:35.435023 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:35.434654    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:36.434665 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:36.434620    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:36.435158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:36.434682    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:37.434203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:37.434169    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:37.434376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:37.434220    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:38.434374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:38.434296    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:38.434374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:38.434353    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:39.434638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:39.434601    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:39.435030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:39.434655    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:40.434618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:40.434587    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 01 20:34:40.434772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:40.434640    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 01 20:34:41.934699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:41.934664    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:41.935463 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:41.935442    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:41.936189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:34:41.936169    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:34:51.629063 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:34:51.629017    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:35:17.454563 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.454527    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:35:17.459666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.459645    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.552267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.552232    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:35:17.556324 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.556303    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.567353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.567330    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:35:17.627015 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.626980    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwxb\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627128    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627195 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627174    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627258    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627367 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627344    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627488 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627431    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627488 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627476    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627635 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627571    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.627696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.627668    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.663583 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.663545    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:35:17.728427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728399    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728432    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.728602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728454    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728518    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728590    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.728817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728646    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728817 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728693    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.728927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728813    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728867    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwxb\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728882    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.728927 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728920    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728950    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728973    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.728992    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729017    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729132 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729113    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729148    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729191    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729222    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729248    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.729322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729281    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pstw\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729485 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729353    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.729485 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.729442    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.731098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.731076    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.731214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.731195    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.784698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.784669    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.784895 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.784873    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwxb\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-d65sv\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:17.830286 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830259    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830305    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830335    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830364    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830430    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pstw\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830714 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830578    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830765 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830751    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830757    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830820    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830835    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830921 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830847    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.830921 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.830907    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.831262 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.831236    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.831366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.831303    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.832887 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.832869    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.833107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.833090    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.886295 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.886268    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pstw\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:17.886409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:17.886388    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rszdb\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:18.074105 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.074017    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:18.169664 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.169632    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:18.553918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.553886    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:35:18.554299 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:35:18.554158    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd379cd_cb94_4dd0_b3a3_2fd36c53df35.slice/crio-08bba42e584d391a05fbbca274724be0896fdf5267392019b6da3fa4f163f52b WatchSource:0}: Error finding container 08bba42e584d391a05fbbca274724be0896fdf5267392019b6da3fa4f163f52b: Status 404 returned error can't find the container with id 08bba42e584d391a05fbbca274724be0896fdf5267392019b6da3fa4f163f52b
Nov 01 20:35:18.556992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.556955    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.557091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.557025    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.557091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.557059    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.595844 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:35:18.595816    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98feb445_a93a_4db0_8130_9b46234b7d4c.slice/crio-5c6b9b580a4d304707055f50e297a7a800a0258cd58545a33a33940ab6954024 WatchSource:0}: Error finding container 5c6b9b580a4d304707055f50e297a7a800a0258cd58545a33a33940ab6954024: Status 404 returned error can't find the container with id 5c6b9b580a4d304707055f50e297a7a800a0258cd58545a33a33940ab6954024
Nov 01 20:35:18.598618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.598578    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.598739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.598656    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.598739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.598717    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:35:18.604525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.603554    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:35:18.831870 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.831766    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" event={"ID":"98feb445-a93a-4db0-8130-9b46234b7d4c","Type":"ContainerStarted","Data":"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"}
Nov 01 20:35:18.831870 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.831834    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" event={"ID":"98feb445-a93a-4db0-8130-9b46234b7d4c","Type":"ContainerStarted","Data":"5c6b9b580a4d304707055f50e297a7a800a0258cd58545a33a33940ab6954024"}
Nov 01 20:35:18.833971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.833934    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" event={"ID":"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35","Type":"ContainerStarted","Data":"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"}
Nov 01 20:35:18.834166 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.833994    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" event={"ID":"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35","Type":"ContainerStarted","Data":"08bba42e584d391a05fbbca274724be0896fdf5267392019b6da3fa4f163f52b"}
Nov 01 20:35:18.952059 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:18.951974    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podStartSLOduration=1.951955895 podStartE2EDuration="1.951955895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:18.946460774 +0000 UTC m=+1972.331358366" watchObservedRunningTime="2026-11-01 20:35:18.951955895 +0000 UTC m=+1972.336853486"
Nov 01 20:35:19.063677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.063614    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podStartSLOduration=2.063593358 podStartE2EDuration="2.063593358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:19.06039982 +0000 UTC m=+1972.445297411" watchObservedRunningTime="2026-11-01 20:35:19.063593358 +0000 UTC m=+1972.448490948"
Nov 01 20:35:19.074476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.074441    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:19.076136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.076106    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:19.076264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.076167    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:19.174571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.171901    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:19.174894 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.174862    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:19.175039 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:19.175016    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:20.082932 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:20.082891    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:20.083540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:20.083509    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:20.173518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:20.171853    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:20.173518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:20.171915    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:21.080517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:21.079904    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:21.080517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:21.079975    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:21.174768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:21.174733    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:21.175325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:21.175296    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:21.647720 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:35:21.647677    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:35:22.075915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:22.075875    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:22.076233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:22.076203    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:22.173915 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:22.173870    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:22.174160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:22.174133    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:23.081517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:23.080953    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:23.081517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:23.081023    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:23.173899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:23.173858    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:23.174181 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:23.174152    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:24.078517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:24.077873    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:24.078517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:24.077935    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:24.171903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:24.171861    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:24.172588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:24.172541    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:25.074640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:25.074603    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:25.074835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:25.074682    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:25.170846 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:25.170803    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:25.171021 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:25.170872    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:26.075869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:26.075397    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 20:35:26.075869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:26.075471    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 20:35:26.169929 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:26.169894    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 01 20:35:26.170123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:26.169943    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 01 20:35:28.196610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.193600    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:35:28.196610 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.193665    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:35:28.231699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.231664    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:35:28.231876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.231742    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:35:28.315761 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.315726    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:28.316399 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.316375    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:28.317078 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.317056    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:35:28.351005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.350970    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:28.351349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.351327    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:28.352248 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:35:28.352116    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:35:51.628685 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:35:51.628650    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:36:05.146355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.146312    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:36:05.146885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.146572    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" podUID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" containerName="maas-api" containerID="cri-o://f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf" gracePeriod=30
Nov 01 20:36:05.439257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.439229    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:36:05.492136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.492106    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\" (UniqueName: \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\") pod \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") "
Nov 01 20:36:05.492314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.492159    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\") pod \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") "
Nov 01 20:36:05.492314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.492199    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") pod \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") "
Nov 01 20:36:05.492446 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.492324    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\") pod \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\" (UID: \"b00046a0-80ff-41aa-b9fc-a9cca32c0b01\") "
Nov 01 20:36:05.492975 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.492946    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca" pod "b00046a0-80ff-41aa-b9fc-a9cca32c0b01" (UID: "b00046a0-80ff-41aa-b9fc-a9cca32c0b01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:36:05.494529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.494469    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls" pod "b00046a0-80ff-41aa-b9fc-a9cca32c0b01" (UID: "b00046a0-80ff-41aa-b9fc-a9cca32c0b01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:36:05.494803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.494780    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs" pod "b00046a0-80ff-41aa-b9fc-a9cca32c0b01" (UID: "b00046a0-80ff-41aa-b9fc-a9cca32c0b01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:36:05.494803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.494791    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr" pod "b00046a0-80ff-41aa-b9fc-a9cca32c0b01" (UID: "b00046a0-80ff-41aa-b9fc-a9cca32c0b01"). InnerVolumeSpecName "kube-api-access-px7zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:05.500074 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.500051    5066 generic.go:356] "Generic (PLEG): container finished" podID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" containerID="f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf" exitCode=0
Nov 01 20:36:05.500188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.500095    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"
Nov 01 20:36:05.500188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.500123    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" event={"ID":"b00046a0-80ff-41aa-b9fc-a9cca32c0b01","Type":"ContainerDied","Data":"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"}
Nov 01 20:36:05.500188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.500174    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf" event={"ID":"b00046a0-80ff-41aa-b9fc-a9cca32c0b01","Type":"ContainerDied","Data":"a381d94a913d8af0ceb9c86895ff399f41528ef7b3029c55e5a99030fb834390"}
Nov 01 20:36:05.500345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.500197    5066 scope.go:122] "RemoveContainer" containerID="f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"
Nov 01 20:36:05.550380 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.550264    5066 scope.go:122] "RemoveContainer" containerID="f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"
Nov 01 20:36:05.556286 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:36:05.556250    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf\": container with ID starting with f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf not found: ID does not exist" containerID="f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"
Nov 01 20:36:05.556521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.556480    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf"} err="failed to get container status \"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf\": rpc error: code = NotFound desc = could not find container \"f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf\": container with ID starting with f290f6e6840472273f7a8be06505143f7ca542539dc343ef380ef38e97b40cdf not found: ID does not exist"
Nov 01 20:36:05.595638 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.595592    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7zr\" (UniqueName: \"kubernetes.io/projected/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-kube-api-access-px7zr\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:05.595822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.595645    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:05.595822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.595702    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:05.595822 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.595728    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00046a0-80ff-41aa-b9fc-a9cca32c0b01-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:05.667884 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.667804    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:36:05.671537 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:05.671462    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f154f7-85fb68558d-qwfqf"]
Nov 01 20:36:06.736035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:06.735999    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" path="/var/lib/kubelet/pods/b00046a0-80ff-41aa-b9fc-a9cca32c0b01/volumes"
Nov 01 20:36:07.930317 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:07.930242    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:36:07.930734 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:07.930431    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" containerID="cri-o://7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f" gracePeriod=30
Nov 01 20:36:08.347271 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:08.347231    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:36:08.347447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:08.347296    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:36:13.541557 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.539559    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:36:13.559038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.559005    5066 generic.go:356] "Generic (PLEG): container finished" podID="c26ffc32-4d83-4037-938e-db94b36b3970" containerID="7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f" exitCode=0
Nov 01 20:36:13.559263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.559052    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"
Nov 01 20:36:13.559263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.559096    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" event={"ID":"c26ffc32-4d83-4037-938e-db94b36b3970","Type":"ContainerDied","Data":"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"}
Nov 01 20:36:13.559263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.559158    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb" event={"ID":"c26ffc32-4d83-4037-938e-db94b36b3970","Type":"ContainerDied","Data":"a69d1e2f714b25148aba28d300c17c548c7287c36fd01fa90c5bb81a8f378821"}
Nov 01 20:36:13.559263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.559190    5066 scope.go:122] "RemoveContainer" containerID="7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"
Nov 01 20:36:13.579740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.579720    5066 scope.go:122] "RemoveContainer" containerID="7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"
Nov 01 20:36:13.610908 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:36:13.610828    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f\": container with ID starting with 7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f not found: ID does not exist" containerID="7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"
Nov 01 20:36:13.610908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.610869    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f"} err="failed to get container status \"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f\": rpc error: code = NotFound desc = could not find container \"7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f\": container with ID starting with 7eaee46b02d52b24c9969e28d24e7e6d7c9189e1758e7da557cec19290a19f2f not found: ID does not exist"
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.647866    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.647926    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648011    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648059    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648095    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648137    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648209    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648264    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.649562 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.648430    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\") pod \"c26ffc32-4d83-4037-938e-db94b36b3970\" (UID: \"c26ffc32-4d83-4037-938e-db94b36b3970\") "
Nov 01 20:36:13.650338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.650283    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:13.651000 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.650922    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:36:13.651415 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.651363    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:13.651805 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.651757    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:13.652652 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.652588    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:13.654509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.654457    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:13.657827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.657780    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:36:13.659073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.659052    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "kube-api-access-lvp52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:13.675042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.671044    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token" pod "c26ffc32-4d83-4037-938e-db94b36b3970" (UID: "c26ffc32-4d83-4037-938e-db94b36b3970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:13.788518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.755334    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.788784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.788757    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.788910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.788895    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.789005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.788993    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.789098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.789086    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26ffc32-4d83-4037-938e-db94b36b3970-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.802560 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.789176    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26ffc32-4d83-4037-938e-db94b36b3970-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.802742 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.802719    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26ffc32-4d83-4037-938e-db94b36b3970-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.802859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.802843    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:13.827514 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:13.802950    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvp52\" (UniqueName: \"kubernetes.io/projected/c26ffc32-4d83-4037-938e-db94b36b3970-kube-api-access-lvp52\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:14.153505 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.153390    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:36:14.156462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.156432    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-d27sb"]
Nov 01 20:36:14.370631 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.370591    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:36:14.370908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.370877    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:36:14.404727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.404608    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:36:14.405027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.404993    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:36:14.760065 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:14.760024    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" path="/var/lib/kubelet/pods/c26ffc32-4d83-4037-938e-db94b36b3970/volumes"
Nov 01 20:36:21.629334 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:36:21.629291    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:36:38.118344 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:38.118303    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:36:38.118860 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:38.118550    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" containerID="cri-o://90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5" gracePeriod=30
Nov 01 20:36:38.410793 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:38.410717    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:36:38.410933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:38.410785    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:36:43.391353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.391299    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:36:43.516984 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.516953    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517002    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517042    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517082    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517149 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517119    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517153    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517186    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517231    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517376 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517299    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\") pod \"0f85f370-eea4-4d59-84fa-3749a5dad918\" (UID: \"0f85f370-eea4-4d59-84fa-3749a5dad918\") "
Nov 01 20:36:43.517596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517384    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:43.517596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517402    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:43.517596 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:43.517762 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517609    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:43.517981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.517961    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.518141 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.518121    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.518254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.518143    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.518254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.518159    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.518254 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.518045    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:36:43.519576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.519547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:43.519676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.519603    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:36:43.519739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.519713    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "kube-api-access-w79xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:36:43.520019 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.520000    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy" pod "0f85f370-eea4-4d59-84fa-3749a5dad918" (UID: "0f85f370-eea4-4d59-84fa-3749a5dad918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:36:43.619217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.619185    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.619217 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.619212    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f85f370-eea4-4d59-84fa-3749a5dad918-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.619435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.619226    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.619435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.619239    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79xp\" (UniqueName: \"kubernetes.io/projected/0f85f370-eea4-4d59-84fa-3749a5dad918-kube-api-access-w79xp\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.619435 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.619249    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f85f370-eea4-4d59-84fa-3749a5dad918-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:36:43.857640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.857610    5066 generic.go:356] "Generic (PLEG): container finished" podID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerID="90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5" exitCode=0
Nov 01 20:36:43.857903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.857656    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"
Nov 01 20:36:43.857903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.857686    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" event={"ID":"0f85f370-eea4-4d59-84fa-3749a5dad918","Type":"ContainerDied","Data":"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"}
Nov 01 20:36:43.857903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.857724    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw" event={"ID":"0f85f370-eea4-4d59-84fa-3749a5dad918","Type":"ContainerDied","Data":"1849d978949ab03488d2905a0a37739372a9ced1d5a1826a7fb48c72d1f2ed0b"}
Nov 01 20:36:43.857903 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.857744    5066 scope.go:122] "RemoveContainer" containerID="90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"
Nov 01 20:36:43.868406 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.868386    5066 scope.go:122] "RemoveContainer" containerID="90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"
Nov 01 20:36:43.868668 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:36:43.868651    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5\": container with ID starting with 90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5 not found: ID does not exist" containerID="90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"
Nov 01 20:36:43.868717 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:43.868674    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5"} err="failed to get container status \"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5\": rpc error: code = NotFound desc = could not find container \"90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5\": container with ID starting with 90f4053ef37ba16cdc69d7da3924f79d9222d3ebdb5de951a52ae8a2ec1fa8e5 not found: ID does not exist"
Nov 01 20:36:44.020663 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:44.020632    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:36:44.070394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:44.070361    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-g22jw"]
Nov 01 20:36:44.736301 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:36:44.736257    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" path="/var/lib/kubelet/pods/0f85f370-eea4-4d59-84fa-3749a5dad918/volumes"
Nov 01 20:36:51.628728 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:36:51.628691    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:37:21.628848 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:37:21.628803    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:37:27.143032 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:37:27.143002    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:37:27.179331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:37:27.179303    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:37:46.754627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:37:46.754550    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:37:51.628627 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:37:51.628591    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:37:55.427727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:37:55.427691    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:37:55.537628 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:37:55.537591    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:38:00.428708 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:00.428666    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing" containerID="cri-o://f160b11353dcd81c3b6e9fdfbc0d39694005bbfdefc6f37838efb49322f69350" gracePeriod=25
Nov 01 20:38:00.538273 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:00.538229    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing" containerID="cri-o://7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b" gracePeriod=25
Nov 01 20:38:01.774611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:01.774572    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main" containerID="cri-o://dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03" gracePeriod=45
Nov 01 20:38:02.051682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.051656    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:38:02.125136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125104    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125207    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125232    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125330    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:02.125409 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125346    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125406    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\" (UniqueName: \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125434    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\") pod \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\" (UID: \"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52\") "
Nov 01 20:38:02.125546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125528    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:02.125712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125689    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:02.125901 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125881    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.125901 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125900    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.126068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.125915    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.127439 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.127407    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:02.127582 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.127469    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:02.127653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.127617    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl" pod "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" (UID: "bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52"). InnerVolumeSpecName "kube-api-access-2sqtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:02.227874 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.227835    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.227874 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.227875    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.228120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.227892    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqtl\" (UniqueName: \"kubernetes.io/projected/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52-kube-api-access-2sqtl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:02.487589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.487553    5066 generic.go:356] "Generic (PLEG): container finished" podID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerID="dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03" exitCode=0
Nov 01 20:38:02.487770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.487597    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"
Nov 01 20:38:02.487770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.487634    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" event={"ID":"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52","Type":"ContainerDied","Data":"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"}
Nov 01 20:38:02.487770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.487689    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn" event={"ID":"bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52","Type":"ContainerDied","Data":"d2add6daf5eab8a15c2f8873f295396a5449fa9a4478fae4e7fc074b321fc000"}
Nov 01 20:38:02.487770 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.487712    5066 scope.go:122] "RemoveContainer" containerID="dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"
Nov 01 20:38:02.500092 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.500068    5066 scope.go:122] "RemoveContainer" containerID="dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"
Nov 01 20:38:02.500354 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:02.500333    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03\": container with ID starting with dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03 not found: ID does not exist" containerID="dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"
Nov 01 20:38:02.500413 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.500362    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03"} err="failed to get container status \"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03\": rpc error: code = NotFound desc = could not find container \"dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03\": container with ID starting with dc1eb8993c2bb253970a97e69b2b86d7e642a215e66f185a5fadb64be3e30b03 not found: ID does not exist"
Nov 01 20:38:02.651214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.651185    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:38:02.699955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.699926    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/distinct-w0-d204f4-kserve-567f66fcb5-lrlpn"]
Nov 01 20:38:02.731961 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:02.731937    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" path="/var/lib/kubelet/pods/bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52/volumes"
Nov 01 20:38:08.998135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:08.998086    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:38:08.998618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:08.998087    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:38:13.859277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859241    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:38:13.859813 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859800    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859814    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859826    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" containerName="maas-api"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859831    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" containerName="maas-api"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859840    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859845    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy"
Nov 01 20:38:13.859857 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859857    5066 cpu_manager.go:414] "containerMap: removing container" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main"
Nov 01 20:38:13.860053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859862    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main"
Nov 01 20:38:13.860053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859941    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26ffc32-4d83-4037-938e-db94b36b3970" containerName="istio-proxy"
Nov 01 20:38:13.860053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859948    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00046a0-80ff-41aa-b9fc-a9cca32c0b01" containerName="maas-api"
Nov 01 20:38:13.860053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859955    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f85f370-eea4-4d59-84fa-3749a5dad918" containerName="istio-proxy"
Nov 01 20:38:13.860053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.859963    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1bd6d1-46a1-4bda-a2bf-d1eb01430f52" containerName="main"
Nov 01 20:38:13.864668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.864650    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:13.919404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.919370    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:38:13.957169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.957131    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:13.957349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.957192    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:13.957349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.957320    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:13.957476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.957347    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7pd\" (UniqueName: \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:13.964984 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:13.964957    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-13cbc9-serving-cert\""
Nov 01 20:38:14.058980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.058947    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.058980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.058981    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7pd\" (UniqueName: \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.059216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.059088    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.059216 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.059119    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.059216 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:14.059173    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-serving-cert: secret "maas-api-e2e-api-b-13cbc9-serving-cert" not found
Nov 01 20:38:14.059373 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:14.059232    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls podName:622f077e-7b4f-46cd-8924-fbbb21735bc0 nodeName:}" failed. No retries permitted until 2026-11-01 20:38:14.559217916 +0000 UTC m=+2147.944115485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls") pod "maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" (UID: "622f077e-7b4f-46cd-8924-fbbb21735bc0") : secret "maas-api-e2e-api-b-13cbc9-serving-cert" not found
Nov 01 20:38:14.059849 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.059825    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.061523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.061488    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.115370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.115301    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7pd\" (UniqueName: \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.573785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.573756    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.576189 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.576163    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") pod \"maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:14.778247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:14.778212    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:15.161146 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:15.161118    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:38:15.162632 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:38:15.162605    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622f077e_7b4f_46cd_8924_fbbb21735bc0.slice/crio-29175678ae301e26f527fe72ec0b12477c222ee24447612422e500e2755d021a WatchSource:0}: Error finding container 29175678ae301e26f527fe72ec0b12477c222ee24447612422e500e2755d021a: Status 404 returned error can't find the container with id 29175678ae301e26f527fe72ec0b12477c222ee24447612422e500e2755d021a
Nov 01 20:38:15.163980 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:15.163960    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:38:15.571995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:15.571947    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" event={"ID":"622f077e-7b4f-46cd-8924-fbbb21735bc0","Type":"ContainerStarted","Data":"29175678ae301e26f527fe72ec0b12477c222ee24447612422e500e2755d021a"}
Nov 01 20:38:16.580056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:16.580018    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" event={"ID":"622f077e-7b4f-46cd-8924-fbbb21735bc0","Type":"ContainerStarted","Data":"734989b6c0387f1a54647b98abbed657f84e0d4e9ee94a11487fc2d03d0a3f1b"}
Nov 01 20:38:16.580561 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:16.580163    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:16.695531 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:16.695444    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" podStartSLOduration=2.783323509 podStartE2EDuration="3.695431085s" totalImagesPullingTime="912.107576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:16.692810877 +0000 UTC m=+2150.077708466" watchObservedRunningTime="2026-11-01 20:38:16.695431085 +0000 UTC m=+2150.080328675"
Nov 01 20:38:18.998630 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:18.998587    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:38:18.999028 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:18.998590    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:38:20.631190 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.631141    5066 generic.go:356] "Generic (PLEG): container finished" podID="faa85fee-1123-495e-98da-022d9b24b602" containerID="f160b11353dcd81c3b6e9fdfbc0d39694005bbfdefc6f37838efb49322f69350" exitCode=0
Nov 01 20:38:20.631683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.631212    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" event={"ID":"faa85fee-1123-495e-98da-022d9b24b602","Type":"ContainerDied","Data":"f160b11353dcd81c3b6e9fdfbc0d39694005bbfdefc6f37838efb49322f69350"}
Nov 01 20:38:20.711835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.711811    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:38:20.809806 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.809781    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:38:20.867165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.867129    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\" (UniqueName: \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\") pod \"faa85fee-1123-495e-98da-022d9b24b602\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") "
Nov 01 20:38:20.867305 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.867251    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\") pod \"faa85fee-1123-495e-98da-022d9b24b602\" (UID: \"faa85fee-1123-495e-98da-022d9b24b602\") "
Nov 01 20:38:20.867683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.867647    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume" pod "faa85fee-1123-495e-98da-022d9b24b602" (UID: "faa85fee-1123-495e-98da-022d9b24b602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:20.867851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.867822    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa85fee-1123-495e-98da-022d9b24b602-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.869326 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.869305    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25" pod "faa85fee-1123-495e-98da-022d9b24b602" (UID: "faa85fee-1123-495e-98da-022d9b24b602"). InnerVolumeSpecName "kube-api-access-xmf25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:20.969088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.969062    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\" (UniqueName: \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\") pod \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") "
Nov 01 20:38:20.969285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.969098    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\") pod \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\" (UID: \"7524e3e6-fe74-44ff-b9ae-e641fd63a54a\") "
Nov 01 20:38:20.969440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.969414    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmf25\" (UniqueName: \"kubernetes.io/projected/faa85fee-1123-495e-98da-022d9b24b602-kube-api-access-xmf25\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.969565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.969534    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume" pod "7524e3e6-fe74-44ff-b9ae-e641fd63a54a" (UID: "7524e3e6-fe74-44ff-b9ae-e641fd63a54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:20.971194 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:20.971172    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9" pod "7524e3e6-fe74-44ff-b9ae-e641fd63a54a" (UID: "7524e3e6-fe74-44ff-b9ae-e641fd63a54a"). InnerVolumeSpecName "kube-api-access-xsxs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:21.071048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.071021    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxs9\" (UniqueName: \"kubernetes.io/projected/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-kube-api-access-xsxs9\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:21.071048 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.071045    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7524e3e6-fe74-44ff-b9ae-e641fd63a54a-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:21.630342 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:21.630303    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:38:21.640362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.640327    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t" event={"ID":"faa85fee-1123-495e-98da-022d9b24b602","Type":"ContainerDied","Data":"44abcc7707b1cadb6ad123e091cf1e24b249a1aa33d406a973420b7a6df2be11"}
Nov 01 20:38:21.640813 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.640364    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"
Nov 01 20:38:21.640813 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.640389    5066 scope.go:122] "RemoveContainer" containerID="f160b11353dcd81c3b6e9fdfbc0d39694005bbfdefc6f37838efb49322f69350"
Nov 01 20:38:21.641828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.641725    5066 generic.go:356] "Generic (PLEG): container finished" podID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerID="7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b" exitCode=0
Nov 01 20:38:21.641828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.641770    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"
Nov 01 20:38:21.641934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.641827    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" event={"ID":"7524e3e6-fe74-44ff-b9ae-e641fd63a54a","Type":"ContainerDied","Data":"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"}
Nov 01 20:38:21.641934 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.641896    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm" event={"ID":"7524e3e6-fe74-44ff-b9ae-e641fd63a54a","Type":"ContainerDied","Data":"4ab40d6c84723e3bee0d936d51f738016448af610d26dedcd8ffbb7f6dbaf65d"}
Nov 01 20:38:21.651203 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.651181    5066 scope.go:122] "RemoveContainer" containerID="7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"
Nov 01 20:38:21.662250 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.662227    5066 scope.go:122] "RemoveContainer" containerID="7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"
Nov 01 20:38:21.662545 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:21.662520    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b\": container with ID starting with 7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b not found: ID does not exist" containerID="7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"
Nov 01 20:38:21.662627 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.662558    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b"} err="failed to get container status \"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b\": rpc error: code = NotFound desc = could not find container \"7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b\": container with ID starting with 7040fea64dfdb9862c7e36a9cf1e3692a16b3b95cab4a6b15f8e65f77a4edd3b not found: ID does not exist"
Nov 01 20:38:21.803997 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.803955    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:38:21.855364 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.855330    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d204f4-77d45ddd55-4m82t"]
Nov 01 20:38:21.961165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.961123    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:38:21.966953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:21.966921    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d204f4-554dbdcf79-gjlcm"]
Nov 01 20:38:22.593188 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:22.593156    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:38:22.733243 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:22.733212    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" path="/var/lib/kubelet/pods/7524e3e6-fe74-44ff-b9ae-e641fd63a54a/volumes"
Nov 01 20:38:22.733660 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:22.733645    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa85fee-1123-495e-98da-022d9b24b602" path="/var/lib/kubelet/pods/faa85fee-1123-495e-98da-022d9b24b602/volumes"
Nov 01 20:38:23.944559 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.944520    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:23.945322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945302    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing"
Nov 01 20:38:23.945419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945323    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing"
Nov 01 20:38:23.945419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945354    5066 cpu_manager.go:414] "containerMap: removing container" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing"
Nov 01 20:38:23.945419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945362    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing"
Nov 01 20:38:23.945615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945532    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7524e3e6-fe74-44ff-b9ae-e641fd63a54a" containerName="payload-pre-processing"
Nov 01 20:38:23.945615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.945549    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa85fee-1123-495e-98da-022d9b24b602" containerName="payload-processing"
Nov 01 20:38:23.950602 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:23.950580    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.030678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.018646    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.030678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.018794    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvx8\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.030678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.020889    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.030678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.020983    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.031462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.031427    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:24.041683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.041652    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.041869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.041769    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.041869 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.041828    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.041994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.041888    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.041994 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.041938    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.142672 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142636    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.142866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142685    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvx8\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.142866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142742    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.142866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142803    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.142866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142847    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143083 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142906    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143083 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.142954    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143083 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143014    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143083 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143058    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143323 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143298    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143361    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143460 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143435    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143453    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.143680 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.143660    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.145105 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.145083    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.145351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.145332    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.200979 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.200901    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.201145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.201019    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvx8\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.264075 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.264042    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:24.695879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.695853    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:24.696898 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:38:24.696867    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1816e3_6546_40b5_a5f9_f1705d590828.slice/crio-6b85906ccfa87f8b2787a2a20dd762095b1553446924fc29d54f298a3627a9d1 WatchSource:0}: Error finding container 6b85906ccfa87f8b2787a2a20dd762095b1553446924fc29d54f298a3627a9d1: Status 404 returned error can't find the container with id 6b85906ccfa87f8b2787a2a20dd762095b1553446924fc29d54f298a3627a9d1
Nov 01 20:38:24.699165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.699136    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:38:24.699230 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.699199    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:38:24.699268 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:24.699231    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 20:38:25.696512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:25.695752    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" event={"ID":"1b1816e3-6546-40b5-a5f9-f1705d590828","Type":"ContainerStarted","Data":"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"}
Nov 01 20:38:25.696512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:25.695861    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" event={"ID":"1b1816e3-6546-40b5-a5f9-f1705d590828","Type":"ContainerStarted","Data":"6b85906ccfa87f8b2787a2a20dd762095b1553446924fc29d54f298a3627a9d1"}
Nov 01 20:38:25.876518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:25.871339    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podStartSLOduration=2.871319722 podStartE2EDuration="2.871319722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:25.867288614 +0000 UTC m=+2159.252186203" watchObservedRunningTime="2026-11-01 20:38:25.871319722 +0000 UTC m=+2159.256217312"
Nov 01 20:38:26.266776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:26.266740    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:26.267160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:26.267133    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:26.267314 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:26.267291    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:27.272517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:27.266788    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:27.272517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:27.266846    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:28.264872 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:28.264838    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:28.265128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:28.265100    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:29.270527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:29.265843    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:29.270527 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:29.265906    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:30.266774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:30.266735    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:30.267080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:30.267052    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:31.265059 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:31.265025    5066 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 20:38:31.265468 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:31.265079    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 20:38:32.361319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:32.361286    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:32.362016 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:32.361985    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:32.364208 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:32.364179    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:41.489414 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:41.489379    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:41.489880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:41.489647    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy" containerID="cri-o://ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc" gracePeriod=30
Nov 01 20:38:41.506155 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:41.506122    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:38:41.506330 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:41.506310    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy" containerID="cri-o://96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09" gracePeriod=30
Nov 01 20:38:44.366640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:44.366596    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:44.367263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:44.367227    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:44.397517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:44.393701    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:44.397517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:44.393772    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:46.807328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.807300    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:46.810480 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.810463    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:38:46.829746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829722    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.829835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829767    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.829835 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829815    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.829941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829845    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.829941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829892    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.829941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829926    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.829973    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830000    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.830102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830052    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.830249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830104    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830147    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830234    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.830249 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830237    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830265    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830343    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830344    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830390    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830428    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830464    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830571    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\") pod \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\" (UID: \"57af0d1d-9857-4cfe-b4f0-04c03f36ee56\") "
Nov 01 20:38:46.830827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830588    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.830827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830606    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\") pod \"1b1816e3-6546-40b5-a5f9-f1705d590828\" (UID: \"1b1816e3-6546-40b5-a5f9-f1705d590828\") "
Nov 01 20:38:46.830827 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.830797    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.831677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.831093    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.831677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.831272    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:46.831677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.831342    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:46.831677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.831617    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.831677 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.831639    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832486    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832538    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832551    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832565    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832578    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832592    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1816e3-6546-40b5-a5f9-f1705d590828-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832607    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832621    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832634    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.832767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.832662    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.833612 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833587    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "kube-api-access-7bvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:46.833702 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833684    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:46.833780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833718    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:46.833780 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833749    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:38:46.833855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833781    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.833855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.833821    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm" pod "57af0d1d-9857-4cfe-b4f0-04c03f36ee56" (UID: "57af0d1d-9857-4cfe-b4f0-04c03f36ee56"). InnerVolumeSpecName "kube-api-access-mdbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:46.834047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.834028    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:46.834101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.834071    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo" pod "1b1816e3-6546-40b5-a5f9-f1705d590828" (UID: "1b1816e3-6546-40b5-a5f9-f1705d590828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:38:46.913594 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.913562    5066 generic.go:356] "Generic (PLEG): container finished" podID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerID="96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09" exitCode=0
Nov 01 20:38:46.913752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.913625    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"
Nov 01 20:38:46.913752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.913629    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" event={"ID":"57af0d1d-9857-4cfe-b4f0-04c03f36ee56","Type":"ContainerDied","Data":"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"}
Nov 01 20:38:46.913752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.913708    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v" event={"ID":"57af0d1d-9857-4cfe-b4f0-04c03f36ee56","Type":"ContainerDied","Data":"75c62a5fc62a27c066c1f2bd18dc5677237579c91ace8efb56e61476a28e6f57"}
Nov 01 20:38:46.913752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.913740    5066 scope.go:122] "RemoveContainer" containerID="96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"
Nov 01 20:38:46.916727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.916702    5066 generic.go:356] "Generic (PLEG): container finished" podID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerID="ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc" exitCode=0
Nov 01 20:38:46.916851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.916747    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" event={"ID":"1b1816e3-6546-40b5-a5f9-f1705d590828","Type":"ContainerDied","Data":"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"}
Nov 01 20:38:46.916851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.916767    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"
Nov 01 20:38:46.916851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.916786    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd" event={"ID":"1b1816e3-6546-40b5-a5f9-f1705d590828","Type":"ContainerDied","Data":"6b85906ccfa87f8b2787a2a20dd762095b1553446924fc29d54f298a3627a9d1"}
Nov 01 20:38:46.925236 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.925214    5066 scope.go:122] "RemoveContainer" containerID="96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"
Nov 01 20:38:46.925543 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:46.925513    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09\": container with ID starting with 96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09 not found: ID does not exist" containerID="96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"
Nov 01 20:38:46.925608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.925553    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09"} err="failed to get container status \"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09\": rpc error: code = NotFound desc = could not find container \"96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09\": container with ID starting with 96fb20a3da7bf5bb042a594b87ff23ad9cc971eaccea366ff8eb1631002e6c09 not found: ID does not exist"
Nov 01 20:38:46.925608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.925575    5066 scope.go:122] "RemoveContainer" containerID="ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"
Nov 01 20:38:46.933830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933809    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933833    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdbvm\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-kube-api-access-mdbvm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933849    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933862    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933876    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933889    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvx8\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-kube-api-access-7bvx8\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933903    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1816e3-6546-40b5-a5f9-f1705d590828-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.933931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.933915    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57af0d1d-9857-4cfe-b4f0-04c03f36ee56-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.935278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.935261    5066 scope.go:122] "RemoveContainer" containerID="ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"
Nov 01 20:38:46.935506 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:46.935474    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc\": container with ID starting with ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc not found: ID does not exist" containerID="ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"
Nov 01 20:38:46.935568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:46.935514    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc"} err="failed to get container status \"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc\": rpc error: code = NotFound desc = could not find container \"ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc\": container with ID starting with ad4b27cc100245dc997c11b1e6787a7f4ba2c999932446174814ee273b2c0dcc not found: ID does not exist"
Nov 01 20:38:47.085812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:47.085673    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:47.096641 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:47.096614    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-qsskd"]
Nov 01 20:38:47.259904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:47.259873    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:38:47.265302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:47.265277    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-jgf4v"]
Nov 01 20:38:48.730668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:48.730623    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" path="/var/lib/kubelet/pods/1b1816e3-6546-40b5-a5f9-f1705d590828/volumes"
Nov 01 20:38:48.731265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:38:48.731244    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" path="/var/lib/kubelet/pods/57af0d1d-9857-4cfe-b4f0-04c03f36ee56/volumes"
Nov 01 20:38:51.628859 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:38:51.628821    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:39:09.140898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:09.140808    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:39:09.141489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:09.141042    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy" containerID="cri-o://272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4" gracePeriod=30
Nov 01 20:39:14.396322 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.396294    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:39:14.442766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442689    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442766 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442750    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442793    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442826    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442879    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442912    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.442976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.442953    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.443233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443006    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.443233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443072    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\") pod \"2462138c-3486-4f81-a5b3-46b2860b3d4d\" (UID: \"2462138c-3486-4f81-a5b3-46b2860b3d4d\") "
Nov 01 20:39:14.443233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443092    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:14.443233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443155    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:14.443233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443178    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:14.443520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443393    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:14.443520 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443448    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:14.443639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443578    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.443639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443592    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.443639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443603    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2462138c-3486-4f81-a5b3-46b2860b3d4d-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.443639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443611    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.443639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.443620    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.444967 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.444947    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "kube-api-access-pjwjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:14.445101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.445088    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:39:14.445471 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.445450    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:14.445471 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.445452    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token" pod "2462138c-3486-4f81-a5b3-46b2860b3d4d" (UID: "2462138c-3486-4f81-a5b3-46b2860b3d4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:14.545170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.545126    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.545170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.545170    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.545393 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.545185    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjwjb\" (UniqueName: \"kubernetes.io/projected/2462138c-3486-4f81-a5b3-46b2860b3d4d-kube-api-access-pjwjb\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:14.545393 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:14.545201    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2462138c-3486-4f81-a5b3-46b2860b3d4d-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:15.102985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.102882    5066 generic.go:356] "Generic (PLEG): container finished" podID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerID="272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4" exitCode=0
Nov 01 20:39:15.102985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.102938    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"
Nov 01 20:39:15.103226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.102975    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" event={"ID":"2462138c-3486-4f81-a5b3-46b2860b3d4d","Type":"ContainerDied","Data":"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"}
Nov 01 20:39:15.103226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.103036    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt" event={"ID":"2462138c-3486-4f81-a5b3-46b2860b3d4d","Type":"ContainerDied","Data":"265a9ba91e64b60f0390a47c7bc6aef594d0222e7675a4fd7cf6fbdae1946cd3"}
Nov 01 20:39:15.103226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.103066    5066 scope.go:122] "RemoveContainer" containerID="272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"
Nov 01 20:39:15.113791 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.113769    5066 scope.go:122] "RemoveContainer" containerID="272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"
Nov 01 20:39:15.114083 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:39:15.114066    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4\": container with ID starting with 272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4 not found: ID does not exist" containerID="272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"
Nov 01 20:39:15.114140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.114091    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4"} err="failed to get container status \"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4\": rpc error: code = NotFound desc = could not find container \"272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4\": container with ID starting with 272a90f93552cb8ebfeb5676753e4adfcea04adf58ec1e4754da0cf4ac3cf0a4 not found: ID does not exist"
Nov 01 20:39:15.329683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.329651    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:39:15.378833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:15.378739    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-vx7dt"]
Nov 01 20:39:16.730948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:16.730912    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" path="/var/lib/kubelet/pods/2462138c-3486-4f81-a5b3-46b2860b3d4d/volumes"
Nov 01 20:39:21.629306 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:39:21.629265    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:39:37.154992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:37.154957    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:39:37.155485 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:37.155160    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" containerID="cri-o://f2c18e6f59d8a79afcaae30947b128b8d21a14e22a3fda388a6e53c162da1260" gracePeriod=30
Nov 01 20:39:37.204649 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:37.204612    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:39:37.204873 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:37.204834    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" containerID="cri-o://22fa1c5179d2a2c45bd0f17df30306269cad030e2ba08ff9c617e44066d2504f" gracePeriod=30
Nov 01 20:39:41.680037 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:41.679994    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:39:41.680661 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:41.680061    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:39:42.081704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.081664    5066 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:39:42.081879 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.081729    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:39:42.332588 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.332475    5066 generic.go:356] "Generic (PLEG): container finished" podID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerID="22fa1c5179d2a2c45bd0f17df30306269cad030e2ba08ff9c617e44066d2504f" exitCode=0
Nov 01 20:39:42.332699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.332522    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" event={"ID":"859b2536-9662-47b2-a4bc-cdd1dc9e7e78","Type":"ContainerDied","Data":"22fa1c5179d2a2c45bd0f17df30306269cad030e2ba08ff9c617e44066d2504f"}
Nov 01 20:39:42.334544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.334517    5066 generic.go:356] "Generic (PLEG): container finished" podID="baf355f0-37ee-48f4-83ce-930dde15737a" containerID="f2c18e6f59d8a79afcaae30947b128b8d21a14e22a3fda388a6e53c162da1260" exitCode=0
Nov 01 20:39:42.334656 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.334579    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" event={"ID":"baf355f0-37ee-48f4-83ce-930dde15737a","Type":"ContainerDied","Data":"f2c18e6f59d8a79afcaae30947b128b8d21a14e22a3fda388a6e53c162da1260"}
Nov 01 20:39:42.470344 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.470316    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:39:42.497246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.497210    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:39:42.514982 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.514951    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.514998    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515017    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515044    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515078    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515100    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515117    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515142    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515192    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515261    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515289    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515298    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515309    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515318    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515340    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515374    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515439    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515589    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515631    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515664    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\") pod \"baf355f0-37ee-48f4-83ce-930dde15737a\" (UID: \"baf355f0-37ee-48f4-83ce-930dde15737a\") "
Nov 01 20:39:42.515970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515705    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\") pod \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\" (UID: \"859b2536-9662-47b2-a4bc-cdd1dc9e7e78\") "
Nov 01 20:39:42.515970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.515685    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:42.516231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516031    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:42.516231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516125    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.516331 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516245    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.516459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516406    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.516459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516427    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516430    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.516459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516446    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516459 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516460    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516473    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516486    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516518    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516553    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baf355f0-37ee-48f4-83ce-930dde15737a-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.516842 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.516821    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.517974 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.517951    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "kube-api-access-6glg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:42.518096 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.517983    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "kube-api-access-mvjtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:42.518390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.518369    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.518390 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.518381    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:39:42.518984 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.518964    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:42.519131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.519116    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:39:42.519209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.519175    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy" pod "859b2536-9662-47b2-a4bc-cdd1dc9e7e78" (UID: "859b2536-9662-47b2-a4bc-cdd1dc9e7e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:39:42.519696 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.519677    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token" pod "baf355f0-37ee-48f4-83ce-930dde15737a" (UID: "baf355f0-37ee-48f4-83ce-930dde15737a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617080    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617107    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glg4\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-kube-api-access-6glg4\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617118    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617127    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617136    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617146    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baf355f0-37ee-48f4-83ce-930dde15737a-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617154    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617159 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617163    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baf355f0-37ee-48f4-83ce-930dde15737a-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617172    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617180    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baf355f0-37ee-48f4-83ce-930dde15737a-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:42.617533 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:42.617188    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjtq\" (UniqueName: \"kubernetes.io/projected/859b2536-9662-47b2-a4bc-cdd1dc9e7e78-kube-api-access-mvjtq\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:39:43.342264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.342235    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"
Nov 01 20:39:43.342264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.342249    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj" event={"ID":"859b2536-9662-47b2-a4bc-cdd1dc9e7e78","Type":"ContainerDied","Data":"e4c76153f3e231b561fddd9babb3bf091ca40053ec798ec44f48463a90245a48"}
Nov 01 20:39:43.342790 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.342316    5066 scope.go:122] "RemoveContainer" containerID="22fa1c5179d2a2c45bd0f17df30306269cad030e2ba08ff9c617e44066d2504f"
Nov 01 20:39:43.343730 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.343691    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2" event={"ID":"baf355f0-37ee-48f4-83ce-930dde15737a","Type":"ContainerDied","Data":"5b9c3d07f2f39efda920f511b2836e3fe81a28355c92b7d4f17189ed3ce3a56c"}
Nov 01 20:39:43.343730 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.343698    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"
Nov 01 20:39:43.354948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.354931    5066 scope.go:122] "RemoveContainer" containerID="f2c18e6f59d8a79afcaae30947b128b8d21a14e22a3fda388a6e53c162da1260"
Nov 01 20:39:43.509986 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.508400    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:39:43.562898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.562859    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-fhmkj"]
Nov 01 20:39:43.624613 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.624574    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:39:43.627412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:43.627379    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-ssbp2"]
Nov 01 20:39:44.730890 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:44.730860    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" path="/var/lib/kubelet/pods/859b2536-9662-47b2-a4bc-cdd1dc9e7e78/volumes"
Nov 01 20:39:44.731302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:44.731288    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" path="/var/lib/kubelet/pods/baf355f0-37ee-48f4-83ce-930dde15737a/volumes"
Nov 01 20:39:51.629027 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:39:51.628988    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:39:51.975759 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:51.975723    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:39:52.228776 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:52.228691    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:39:56.976453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:56.976407    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing" containerID="cri-o://e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8" gracePeriod=25
Nov 01 20:39:57.229247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:39:57.229149    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing" containerID="cri-o://b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585" gracePeriod=25
Nov 01 20:40:00.207795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.207757    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:40:00.208269 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.207967    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" podUID="622f077e-7b4f-46cd-8924-fbbb21735bc0" containerName="maas-api" containerID="cri-o://734989b6c0387f1a54647b98abbed657f84e0d4e9ee94a11487fc2d03d0a3f1b" gracePeriod=30
Nov 01 20:40:00.453676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.453646    5066 generic.go:356] "Generic (PLEG): container finished" podID="622f077e-7b4f-46cd-8924-fbbb21735bc0" containerID="734989b6c0387f1a54647b98abbed657f84e0d4e9ee94a11487fc2d03d0a3f1b" exitCode=0
Nov 01 20:40:00.453816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.453724    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" event={"ID":"622f077e-7b4f-46cd-8924-fbbb21735bc0","Type":"ContainerDied","Data":"734989b6c0387f1a54647b98abbed657f84e0d4e9ee94a11487fc2d03d0a3f1b"}
Nov 01 20:40:00.480233 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.480209    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:40:00.624631 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.624586    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\") pod \"622f077e-7b4f-46cd-8924-fbbb21735bc0\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") "
Nov 01 20:40:00.624836 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.624715    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") pod \"622f077e-7b4f-46cd-8924-fbbb21735bc0\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") "
Nov 01 20:40:00.624916 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.624837    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\" (UniqueName: \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\") pod \"622f077e-7b4f-46cd-8924-fbbb21735bc0\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") "
Nov 01 20:40:00.625042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.624929    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\") pod \"622f077e-7b4f-46cd-8924-fbbb21735bc0\" (UID: \"622f077e-7b4f-46cd-8924-fbbb21735bc0\") "
Nov 01 20:40:00.625339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.625309    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca" pod "622f077e-7b4f-46cd-8924-fbbb21735bc0" (UID: "622f077e-7b4f-46cd-8924-fbbb21735bc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:00.627589 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.627563    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd" pod "622f077e-7b4f-46cd-8924-fbbb21735bc0" (UID: "622f077e-7b4f-46cd-8924-fbbb21735bc0"). InnerVolumeSpecName "kube-api-access-dp7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:00.631941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.631897    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs" pod "622f077e-7b4f-46cd-8924-fbbb21735bc0" (UID: "622f077e-7b4f-46cd-8924-fbbb21735bc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:00.633199 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.633175    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls" pod "622f077e-7b4f-46cd-8924-fbbb21735bc0" (UID: "622f077e-7b4f-46cd-8924-fbbb21735bc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:00.725845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.725815    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7pd\" (UniqueName: \"kubernetes.io/projected/622f077e-7b4f-46cd-8924-fbbb21735bc0-kube-api-access-dp7pd\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:00.725845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.725841    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622f077e-7b4f-46cd-8924-fbbb21735bc0-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:00.725845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.725850    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:00.726152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:00.725861    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622f077e-7b4f-46cd-8924-fbbb21735bc0-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:01.460681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:01.460645    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"
Nov 01 20:40:01.460681 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:01.460661    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf" event={"ID":"622f077e-7b4f-46cd-8924-fbbb21735bc0","Type":"ContainerDied","Data":"29175678ae301e26f527fe72ec0b12477c222ee24447612422e500e2755d021a"}
Nov 01 20:40:01.461134 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:01.460722    5066 scope.go:122] "RemoveContainer" containerID="734989b6c0387f1a54647b98abbed657f84e0d4e9ee94a11487fc2d03d0a3f1b"
Nov 01 20:40:01.623851 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:01.623806    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:40:01.673260 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:01.673229    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-13cbc9-88b5d766f-ts4wf"]
Nov 01 20:40:02.730124 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:02.730094    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="622f077e-7b4f-46cd-8924-fbbb21735bc0" path="/var/lib/kubelet/pods/622f077e-7b4f-46cd-8924-fbbb21735bc0/volumes"
Nov 01 20:40:06.050608 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:06.050568    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:40:06.051056 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:06.050613    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:40:16.050431 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:16.050388    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:40:16.050946 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:16.050579    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:40:17.305074 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.305051    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:40:17.414794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.414755    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\") pod \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") "
Nov 01 20:40:17.414941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.414846    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\" (UniqueName: \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\") pod \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\" (UID: \"ca2a8edc-5560-4f44-88c9-570d0a8a5b95\") "
Nov 01 20:40:17.415251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.415183    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume" pod "ca2a8edc-5560-4f44-88c9-570d0a8a5b95" (UID: "ca2a8edc-5560-4f44-88c9-570d0a8a5b95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:17.415366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.415262    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:17.422516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.422459    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth" pod "ca2a8edc-5560-4f44-88c9-570d0a8a5b95" (UID: "ca2a8edc-5560-4f44-88c9-570d0a8a5b95"). InnerVolumeSpecName "kube-api-access-6dhth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:17.466576 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.466545    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:40:17.516540 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.516456    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhth\" (UniqueName: \"kubernetes.io/projected/ca2a8edc-5560-4f44-88c9-570d0a8a5b95-kube-api-access-6dhth\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:17.558861 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.558828    5066 generic.go:356] "Generic (PLEG): container finished" podID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerID="e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8" exitCode=0
Nov 01 20:40:17.558996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.558868    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"
Nov 01 20:40:17.558996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.558908    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" event={"ID":"ca2a8edc-5560-4f44-88c9-570d0a8a5b95","Type":"ContainerDied","Data":"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"}
Nov 01 20:40:17.558996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.558954    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh" event={"ID":"ca2a8edc-5560-4f44-88c9-570d0a8a5b95","Type":"ContainerDied","Data":"0078228cb08a854edd8a2daf6b10dcc2c6b99a703a610689e09975885aa3fc68"}
Nov 01 20:40:17.558996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.558979    5066 scope.go:122] "RemoveContainer" containerID="e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"
Nov 01 20:40:17.560313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.560287    5066 generic.go:356] "Generic (PLEG): container finished" podID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerID="b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585" exitCode=0
Nov 01 20:40:17.560427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.560323    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"
Nov 01 20:40:17.560427 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.560378    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" event={"ID":"2b5d1eb2-eaae-4806-a983-5125ddc6b411","Type":"ContainerDied","Data":"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"}
Nov 01 20:40:17.560552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.560427    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv" event={"ID":"2b5d1eb2-eaae-4806-a983-5125ddc6b411","Type":"ContainerDied","Data":"e442e9c01b6e18fc3e3cf8c19e87ae49c791b32a2d15936286dfae865427eeb6"}
Nov 01 20:40:17.568937 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.568921    5066 scope.go:122] "RemoveContainer" containerID="e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"
Nov 01 20:40:17.569175 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:40:17.569157    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8\": container with ID starting with e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8 not found: ID does not exist" containerID="e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"
Nov 01 20:40:17.569251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.569185    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8"} err="failed to get container status \"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8\": rpc error: code = NotFound desc = could not find container \"e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8\": container with ID starting with e75ab09220c1fec2c5f7713c7a23357409c4cf964a0ff19dba1245dbfac76fc8 not found: ID does not exist"
Nov 01 20:40:17.569251 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.569206    5066 scope.go:122] "RemoveContainer" containerID="b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"
Nov 01 20:40:17.577235 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.577218    5066 scope.go:122] "RemoveContainer" containerID="b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"
Nov 01 20:40:17.577471 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:40:17.577452    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585\": container with ID starting with b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585 not found: ID does not exist" containerID="b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"
Nov 01 20:40:17.577549 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.577477    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585"} err="failed to get container status \"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585\": rpc error: code = NotFound desc = could not find container \"b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585\": container with ID starting with b175d28f8a73a0ed1378ce0592837ae38f713678dc88f457b9d200976d365585 not found: ID does not exist"
Nov 01 20:40:17.617834 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.617806    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\" (UniqueName: \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\") pod \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") "
Nov 01 20:40:17.617955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.617881    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\") pod \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\" (UID: \"2b5d1eb2-eaae-4806-a983-5125ddc6b411\") "
Nov 01 20:40:17.618231 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.618210    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume" pod "2b5d1eb2-eaae-4806-a983-5125ddc6b411" (UID: "2b5d1eb2-eaae-4806-a983-5125ddc6b411"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:17.618382 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.618366    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5d1eb2-eaae-4806-a983-5125ddc6b411-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:17.619795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.619773    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl" pod "2b5d1eb2-eaae-4806-a983-5125ddc6b411" (UID: "2b5d1eb2-eaae-4806-a983-5125ddc6b411"). InnerVolumeSpecName "kube-api-access-c7pdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:17.719614 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.719576    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7pdl\" (UniqueName: \"kubernetes.io/projected/2b5d1eb2-eaae-4806-a983-5125ddc6b411-kube-api-access-c7pdl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:17.742101 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.742073    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:40:17.747736 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.747711    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-1b1134-68f67c79c-mgsqh"]
Nov 01 20:40:17.980320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.980285    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:40:17.986142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:17.986114    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1b1134-6bb5988d6c-mzqlv"]
Nov 01 20:40:18.730700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:18.730664    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" path="/var/lib/kubelet/pods/2b5d1eb2-eaae-4806-a983-5125ddc6b411/volumes"
Nov 01 20:40:18.731081 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:18.731057    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" path="/var/lib/kubelet/pods/ca2a8edc-5560-4f44-88c9-570d0a8a5b95/volumes"
Nov 01 20:40:21.637988 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:40:21.637933    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:40:25.600137 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.600095    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:40:25.600640 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.600374    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" podUID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" containerName="maas-api" containerID="cri-o://b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c" gracePeriod=30
Nov 01 20:40:25.872803 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.872777    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:40:25.930831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.930798    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\") pod \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") "
Nov 01 20:40:25.930984 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.930926    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\" (UniqueName: \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\") pod \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") "
Nov 01 20:40:25.931030 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.930998    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\") pod \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") "
Nov 01 20:40:25.931079 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.931039    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") pod \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\" (UID: \"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5\") "
Nov 01 20:40:25.931416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.931382    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca" pod "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" (UID: "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:25.932883 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.932855    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs" pod "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" (UID: "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:25.933027 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.933011    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws" pod "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" (UID: "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5"). InnerVolumeSpecName "kube-api-access-z99ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:25.933087 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:25.933073    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls" pod "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" (UID: "c4e04fb1-1e80-44d7-8b7e-576cf02f92c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:26.032643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.032609    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:26.032643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.032638    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:26.032643 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.032649    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:26.032872 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.032658    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99ws\" (UniqueName: \"kubernetes.io/projected/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5-kube-api-access-z99ws\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:40:26.631170 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.631133    5066 generic.go:356] "Generic (PLEG): container finished" podID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" containerID="b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c" exitCode=0
Nov 01 20:40:26.631604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.631180    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"
Nov 01 20:40:26.631604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.631197    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" event={"ID":"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5","Type":"ContainerDied","Data":"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"}
Nov 01 20:40:26.631604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.631242    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh" event={"ID":"c4e04fb1-1e80-44d7-8b7e-576cf02f92c5","Type":"ContainerDied","Data":"d88468c46a5fb38db40d577e548ecd5229cede7335c09588482f749aa29ad4ec"}
Nov 01 20:40:26.631604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.631263    5066 scope.go:122] "RemoveContainer" containerID="b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"
Nov 01 20:40:26.642140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.642120    5066 scope.go:122] "RemoveContainer" containerID="b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"
Nov 01 20:40:26.642433 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:40:26.642404    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c\": container with ID starting with b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c not found: ID does not exist" containerID="b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"
Nov 01 20:40:26.642517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.642444    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c"} err="failed to get container status \"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c\": rpc error: code = NotFound desc = could not find container \"b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c\": container with ID starting with b2b9081bfbce2fd5bb05f3d5b97c8b3d7000d711fca9f5ab0e594488864b9d6c not found: ID does not exist"
Nov 01 20:40:26.795756 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.795723    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:40:26.845120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:26.845085    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1b1134-7b8457cd7d-tlhfh"]
Nov 01 20:40:28.731161 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:40:28.731119    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" path="/var/lib/kubelet/pods/c4e04fb1-1e80-44d7-8b7e-576cf02f92c5/volumes"
Nov 01 20:40:51.629964 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:40:51.629876    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:41:21.627926 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:41:21.627890    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:41:40.264168 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.264134    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:41:40.264699 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.264318    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerName="discovery" containerID="cri-o://2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441" gracePeriod=30
Nov 01 20:41:40.522345 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.522285    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:41:40.640447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640412    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640545    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640601    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640629    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640719    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640746    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.640826 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.640780    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\") pod \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\" (UID: \"ccc9a9a4-0a0f-4499-9701-9129ad9b573d\") "
Nov 01 20:41:40.641795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.641761    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:40.643761 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.643734    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:40.643885 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.643866    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:40.643995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.643966    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "kube-api-access-rbc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:40.645624 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.645600    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:40.652943 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.652917    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:40.653038 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.653016    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts" pod "ccc9a9a4-0a0f-4499-9701-9129ad9b573d" (UID: "ccc9a9a4-0a0f-4499-9701-9129ad9b573d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:40.742637 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742609    5066 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-ca-configmap\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742637 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742636    5066 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-kubeconfig\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742651    5066 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-cacerts\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742662    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbc55\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-kube-api-access-rbc55\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742675    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742688    5066 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-local-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:40.742787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:40.742699    5066 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ccc9a9a4-0a0f-4499-9701-9129ad9b573d-istio-csr-dns-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:41:41.088785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.088745    5066 generic.go:356] "Generic (PLEG): container finished" podID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerID="2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441" exitCode=0
Nov 01 20:41:41.088960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.088796    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"
Nov 01 20:41:41.088960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.088835    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" event={"ID":"ccc9a9a4-0a0f-4499-9701-9129ad9b573d","Type":"ContainerDied","Data":"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"}
Nov 01 20:41:41.088960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.088891    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz" event={"ID":"ccc9a9a4-0a0f-4499-9701-9129ad9b573d","Type":"ContainerDied","Data":"458482adaa5a5a18f9345c7a8e312c8200fffb134d27a2ae6a56a9cd1f580134"}
Nov 01 20:41:41.088960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.088911    5066 scope.go:122] "RemoveContainer" containerID="2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"
Nov 01 20:41:41.098587 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.098564    5066 scope.go:122] "RemoveContainer" containerID="2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"
Nov 01 20:41:41.098842 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:41:41.098817    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441\": container with ID starting with 2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441 not found: ID does not exist" containerID="2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"
Nov 01 20:41:41.098933 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.098851    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441"} err="failed to get container status \"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441\": rpc error: code = NotFound desc = could not find container \"2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441\": container with ID starting with 2996ae80a7b0be49cb34ace983f7bc28497a0a28e30e2c152bc0e2cf0acd5441 not found: ID does not exist"
Nov 01 20:41:41.252278 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.252245    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:41:41.255062 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:41.255030    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p6kz"]
Nov 01 20:41:42.730429 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:41:42.730392    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" path="/var/lib/kubelet/pods/ccc9a9a4-0a0f-4499-9701-9129ad9b573d/volumes"
Nov 01 20:41:51.629273 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:41:51.629227    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:42:21.628910 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:42:21.628873    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:42:27.181160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:42:27.181060    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:42:27.222877 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:42:27.222853    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:42:51.628044 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:42:51.628003    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:43:19.836726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.836686    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:43:19.837369 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837349    5066 cpu_manager.go:414] "containerMap: removing container" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837372    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837401    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837412    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837434    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" containerName="maas-api"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837441    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" containerName="maas-api"
Nov 01 20:43:19.837462 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837457    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837464    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837477    5066 cpu_manager.go:414] "containerMap: removing container" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837484    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837515    5066 cpu_manager.go:414] "containerMap: removing container" podUID="622f077e-7b4f-46cd-8924-fbbb21735bc0" containerName="maas-api"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837524    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622f077e-7b4f-46cd-8924-fbbb21735bc0" containerName="maas-api"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837537    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837544    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837558    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837565    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837582    5066 cpu_manager.go:414] "containerMap: removing container" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837588    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837610    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerName="discovery"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837617    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerName="discovery"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837744    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2462138c-3486-4f81-a5b3-46b2860b3d4d" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837758    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="859b2536-9662-47b2-a4bc-cdd1dc9e7e78" containerName="istio-proxy"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837769    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="622f077e-7b4f-46cd-8924-fbbb21735bc0" containerName="maas-api"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837777    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5d1eb2-eaae-4806-a983-5125ddc6b411" containerName="payload-pre-processing"
Nov 01 20:43:19.837778 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837786    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2a8edc-5560-4f44-88c9-570d0a8a5b95" containerName="payload-processing"
Nov 01 20:43:19.838607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837795    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="57af0d1d-9857-4cfe-b4f0-04c03f36ee56" containerName="istio-proxy"
Nov 01 20:43:19.838607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837804    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc9a9a4-0a0f-4499-9701-9129ad9b573d" containerName="discovery"
Nov 01 20:43:19.838607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837814    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf355f0-37ee-48f4-83ce-930dde15737a" containerName="istio-proxy"
Nov 01 20:43:19.838607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837826    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1816e3-6546-40b5-a5f9-f1705d590828" containerName="istio-proxy"
Nov 01 20:43:19.838607 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.837838    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e04fb1-1e80-44d7-8b7e-576cf02f92c5" containerName="maas-api"
Nov 01 20:43:19.841370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.841349    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:19.896085 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.896051    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:43:19.942489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.942412    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-cd18c1528ab8\""
Nov 01 20:43:19.942489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.942418    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-e385637602d5\""
Nov 01 20:43:19.942489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.942452    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-4bc05b0bda98\""
Nov 01 20:43:19.942489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.942463    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\""
Nov 01 20:43:19.942752 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.942527    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-t8dl8\""
Nov 01 20:43:19.943718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.943702    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\""
Nov 01 20:43:19.943799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.943721    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\""
Nov 01 20:43:19.943799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:19.943727    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\""
Nov 01 20:43:20.022987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.022955    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfg7q\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.023169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.023010    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.023169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.023137    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.023304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.023243    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.124146 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124110    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.124351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124189    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg7q\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.124351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124240    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.124351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124285    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.124908 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124878    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.125003 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.124943    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.126896 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.126876    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.180091 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.180065    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg7q\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.454381 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.454350    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:20.833312 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:20.833280    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:43:20.835455 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:43:20.835426    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3ae0bb_1291_4ff4_a118_ffc791f34096.slice/crio-a6f995edae7b0b396e7b1a0120c4af2d0d585a4ecf9d65a219dafb4e977857c2 WatchSource:0}: Error finding container a6f995edae7b0b396e7b1a0120c4af2d0d585a4ecf9d65a219dafb4e977857c2: Status 404 returned error can't find the container with id a6f995edae7b0b396e7b1a0120c4af2d0d585a4ecf9d65a219dafb4e977857c2
Nov 01 20:43:21.629184 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:43:21.629149    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:43:21.675772 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:21.675739    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" event={"ID":"bf3ae0bb-1291-4ff4-a118-ffc791f34096","Type":"ContainerStarted","Data":"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"}
Nov 01 20:43:21.675972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:21.675792    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" event={"ID":"bf3ae0bb-1291-4ff4-a118-ffc791f34096","Type":"ContainerStarted","Data":"a6f995edae7b0b396e7b1a0120c4af2d0d585a4ecf9d65a219dafb4e977857c2"}
Nov 01 20:43:21.675972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:21.675863    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:43:21.788854 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:21.788790    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podStartSLOduration=2.7887712970000003 podStartE2EDuration="2.788771297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:43:21.788569374 +0000 UTC m=+2455.173466965" watchObservedRunningTime="2026-11-01 20:43:21.788771297 +0000 UTC m=+2455.173668889"
Nov 01 20:43:32.685995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:32.685948    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:43:42.684722 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:42.684633    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:43:51.628103 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:43:51.628066    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:43:52.684657 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:43:52.684613    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:02.685154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:02.685110    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:12.684445 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:12.684404    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:17.925475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:17.925433    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:44:17.926823 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:17.926795    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:17.974447 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:17.974410    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:17.978318 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:17.978298    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.038172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.038138    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:18.120232 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.120192    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsv2h\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.120419 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.120327    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.120466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.120420    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.120548 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.120465    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.221985 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.221958    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.222412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.221998    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.222412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.222100    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsv2h\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.222412 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.222175    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.222821 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.222793    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.222959 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.222937    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.224777 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.224753    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.309872 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.308308    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsv2h\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.591441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.591347    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:18.987163 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.987132    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:18.988947 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:44:18.988923    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162afbbc_c951_45a3_a6fb_441ff0c6e05e.slice/crio-45b5cc6f080c9da02c7742ad0427061b5268452d88baeddaf1f23575ecc4a37d WatchSource:0}: Error finding container 45b5cc6f080c9da02c7742ad0427061b5268452d88baeddaf1f23575ecc4a37d: Status 404 returned error can't find the container with id 45b5cc6f080c9da02c7742ad0427061b5268452d88baeddaf1f23575ecc4a37d
Nov 01 20:44:18.999167 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:18.999139    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" event={"ID":"162afbbc-c951-45a3-a6fb-441ff0c6e05e","Type":"ContainerStarted","Data":"45b5cc6f080c9da02c7742ad0427061b5268452d88baeddaf1f23575ecc4a37d"}
Nov 01 20:44:20.006754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:20.006716    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" event={"ID":"162afbbc-c951-45a3-a6fb-441ff0c6e05e","Type":"ContainerStarted","Data":"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"}
Nov 01 20:44:20.007171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:20.006838    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:20.122593 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:20.122538    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" podStartSLOduration=3.122524483 podStartE2EDuration="3.122524483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:20.119732242 +0000 UTC m=+2513.504629827" watchObservedRunningTime="2026-11-01 20:44:20.122524483 +0000 UTC m=+2513.507422069"
Nov 01 20:44:21.630190 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:21.630153    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:44:22.926639 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:22.926596    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing" containerID="cri-o://0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f" gracePeriod=25
Nov 01 20:44:23.190935 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.190879    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:44:23.278792 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.278762    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\") pod \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") "
Nov 01 20:44:23.278971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.278799    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\") pod \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") "
Nov 01 20:44:23.278971 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.278890    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\") pod \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") "
Nov 01 20:44:23.279095 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.279069    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\") pod \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\" (UID: \"bf3ae0bb-1291-4ff4-a118-ffc791f34096\") "
Nov 01 20:44:23.279368 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.279339    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume" pod "bf3ae0bb-1291-4ff4-a118-ffc791f34096" (UID: "bf3ae0bb-1291-4ff4-a118-ffc791f34096"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:23.279466 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.279406    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume" pod "bf3ae0bb-1291-4ff4-a118-ffc791f34096" (UID: "bf3ae0bb-1291-4ff4-a118-ffc791f34096"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:23.279621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.279602    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:23.279676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.279627    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3ae0bb-1291-4ff4-a118-ffc791f34096-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:23.280955 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.280935    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials" pod "bf3ae0bb-1291-4ff4-a118-ffc791f34096" (UID: "bf3ae0bb-1291-4ff4-a118-ffc791f34096"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:23.281350 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.281333    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q" pod "bf3ae0bb-1291-4ff4-a118-ffc791f34096" (UID: "bf3ae0bb-1291-4ff4-a118-ffc791f34096"). InnerVolumeSpecName "kube-api-access-dfg7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:23.380621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.380587    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfg7q\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-kube-api-access-dfg7q\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:23.380621 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:23.380616    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3ae0bb-1291-4ff4-a118-ffc791f34096-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:24.030108 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.030074    5066 generic.go:356] "Generic (PLEG): container finished" podID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerID="0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f" exitCode=0
Nov 01 20:44:24.030525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.030121    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"
Nov 01 20:44:24.030525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.030153    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" event={"ID":"bf3ae0bb-1291-4ff4-a118-ffc791f34096","Type":"ContainerDied","Data":"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"}
Nov 01 20:44:24.030525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.030206    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v" event={"ID":"bf3ae0bb-1291-4ff4-a118-ffc791f34096","Type":"ContainerDied","Data":"a6f995edae7b0b396e7b1a0120c4af2d0d585a4ecf9d65a219dafb4e977857c2"}
Nov 01 20:44:24.030525 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.030233    5066 scope.go:122] "RemoveContainer" containerID="0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"
Nov 01 20:44:24.040353 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.040338    5066 scope.go:122] "RemoveContainer" containerID="0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"
Nov 01 20:44:24.040664 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:24.040647    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f\": container with ID starting with 0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f not found: ID does not exist" containerID="0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"
Nov 01 20:44:24.040739 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.040672    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f"} err="failed to get container status \"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f\": rpc error: code = NotFound desc = could not find container \"0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f\": container with ID starting with 0c1de1159049569c19eac9397d167476887b94c6449548b78fcb1ddf3b6f679f not found: ID does not exist"
Nov 01 20:44:24.192820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.192789    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:44:24.243663 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.243629    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5d4ql4v"]
Nov 01 20:44:24.730998 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:24.730969    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" path="/var/lib/kubelet/pods/bf3ae0bb-1291-4ff4-a118-ffc791f34096/volumes"
Nov 01 20:44:30.823685 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.823649    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:30.826898 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.826863    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:30.850267 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.850236    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:30.850760 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.850746    5066 cpu_manager.go:414] "containerMap: removing container" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing"
Nov 01 20:44:30.850808 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.850762    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing"
Nov 01 20:44:30.850854 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.850845    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3ae0bb-1291-4ff4-a118-ffc791f34096" containerName="payload-processing"
Nov 01 20:44:30.855351 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.855334    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.872429 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.872402    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.872591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.872460    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.872591 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.872574    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clfd\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.872698 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.872622    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.911953 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.911923    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:30.976544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.974124    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.976544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.974207    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clfd\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.976544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.974267    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.976544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.974437    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.976544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.975798    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.977308 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.977237    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:30.980392 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:30.980367    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:31.032789 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:31.032759    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clfd\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:31.168831 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:31.168719    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:31.546219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:31.546186    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:31.547693 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:44:31.547665    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8310c81d_3e39_42d9_85db_f1445a1b2112.slice/crio-bb8f2dd9a80894c4d216aa94cbe19c66e10ec3777b5a644a8fae0d8780e230c2 WatchSource:0}: Error finding container bb8f2dd9a80894c4d216aa94cbe19c66e10ec3777b5a644a8fae0d8780e230c2: Status 404 returned error can't find the container with id bb8f2dd9a80894c4d216aa94cbe19c66e10ec3777b5a644a8fae0d8780e230c2
Nov 01 20:44:32.081700 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:32.081662    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" event={"ID":"8310c81d-3e39-42d9-85db-f1445a1b2112","Type":"ContainerStarted","Data":"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"}
Nov 01 20:44:32.082109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:32.081723    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" event={"ID":"8310c81d-3e39-42d9-85db-f1445a1b2112","Type":"ContainerStarted","Data":"bb8f2dd9a80894c4d216aa94cbe19c66e10ec3777b5a644a8fae0d8780e230c2"}
Nov 01 20:44:32.082109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:32.081878    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:32.197087 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:32.197029    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" podStartSLOduration=2.197013595 podStartE2EDuration="2.197013595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:32.195144345 +0000 UTC m=+2525.580041934" watchObservedRunningTime="2026-11-01 20:44:32.197013595 +0000 UTC m=+2525.581911223"
Nov 01 20:44:35.824532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:35.824461    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerName="payload-processing" containerID="cri-o://f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d" gracePeriod=25
Nov 01 20:44:36.070595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.070567    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:36.113785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.113712    5066 generic.go:356] "Generic (PLEG): container finished" podID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerID="f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d" exitCode=0
Nov 01 20:44:36.113785 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.113755    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"
Nov 01 20:44:36.113931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.113786    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" event={"ID":"162afbbc-c951-45a3-a6fb-441ff0c6e05e","Type":"ContainerDied","Data":"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"}
Nov 01 20:44:36.113931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.113881    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56" event={"ID":"162afbbc-c951-45a3-a6fb-441ff0c6e05e","Type":"ContainerDied","Data":"45b5cc6f080c9da02c7742ad0427061b5268452d88baeddaf1f23575ecc4a37d"}
Nov 01 20:44:36.113931 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.113903    5066 scope.go:122] "RemoveContainer" containerID="f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"
Nov 01 20:44:36.123118 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.123099    5066 scope.go:122] "RemoveContainer" containerID="f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"
Nov 01 20:44:36.123397 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:36.123378    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d\": container with ID starting with f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d not found: ID does not exist" containerID="f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"
Nov 01 20:44:36.123449 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.123404    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d"} err="failed to get container status \"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d\": rpc error: code = NotFound desc = could not find container \"f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d\": container with ID starting with f5260adf351a61e49f2b5eaa25113045f1370b17104b6befa4ce42ef8b681d4d not found: ID does not exist"
Nov 01 20:44:36.135782 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.135761    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\") pod \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") "
Nov 01 20:44:36.135909 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.135895    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\") pod \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") "
Nov 01 20:44:36.135978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.135935    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\") pod \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") "
Nov 01 20:44:36.135978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.135971    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\") pod \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\" (UID: \"162afbbc-c951-45a3-a6fb-441ff0c6e05e\") "
Nov 01 20:44:36.136171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.136146    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume" pod "162afbbc-c951-45a3-a6fb-441ff0c6e05e" (UID: "162afbbc-c951-45a3-a6fb-441ff0c6e05e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:36.136472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.136432    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume" pod "162afbbc-c951-45a3-a6fb-441ff0c6e05e" (UID: "162afbbc-c951-45a3-a6fb-441ff0c6e05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:36.136472 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.136451    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:36.137987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.137962    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h" pod "162afbbc-c951-45a3-a6fb-441ff0c6e05e" (UID: "162afbbc-c951-45a3-a6fb-441ff0c6e05e"). InnerVolumeSpecName "kube-api-access-jsv2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:36.138263 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.138243    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials" pod "162afbbc-c951-45a3-a6fb-441ff0c6e05e" (UID: "162afbbc-c951-45a3-a6fb-441ff0c6e05e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:36.237246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.237211    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsv2h\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-kube-api-access-jsv2h\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:36.237246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.237243    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/162afbbc-c951-45a3-a6fb-441ff0c6e05e-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:36.237246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.237253    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162afbbc-c951-45a3-a6fb-441ff0c6e05e-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:36.535007 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.534969    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:36.537833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.537805    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d45tcs56"]
Nov 01 20:44:36.730474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:36.730433    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" path="/var/lib/kubelet/pods/162afbbc-c951-45a3-a6fb-441ff0c6e05e/volumes"
Nov 01 20:44:39.390801 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:39.390759    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:44:39.391284 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:39.391057    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy" containerID="cri-o://d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80" gracePeriod=30
Nov 01 20:44:43.091848 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:43.091803    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:44.658316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.658259    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:44:44.728008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.727954    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.728889    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.728953    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.728996    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.729071    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.729112    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.729147    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.729277    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.729474 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.729323    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\") pod \"d828bb2f-3026-47e9-a96b-53b13430f9bf\" (UID: \"d828bb2f-3026-47e9-a96b-53b13430f9bf\") "
Nov 01 20:44:44.734523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.730901    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "kube-api-access-kkjm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:44.734523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.733870    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:44.735165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.735000    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:44.735275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.735192    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:44.735275 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.735208    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:44.735410 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.735386    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:44.735577 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.735553    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:44.741158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.741136    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:44:44.741566 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.741547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token" pod "d828bb2f-3026-47e9-a96b-53b13430f9bf" (UID: "d828bb2f-3026-47e9-a96b-53b13430f9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832328    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832358    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjm4\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-kube-api-access-kkjm4\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832374    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832387    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832400    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832412    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d828bb2f-3026-47e9-a96b-53b13430f9bf-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832425    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832438    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:44.832552 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:44.832450    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d828bb2f-3026-47e9-a96b-53b13430f9bf-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:45.169689 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.169653    5066 generic.go:356] "Generic (PLEG): container finished" podID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerID="d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80" exitCode=0
Nov 01 20:44:45.169862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.169729    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"
Nov 01 20:44:45.169862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.169739    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" event={"ID":"d828bb2f-3026-47e9-a96b-53b13430f9bf","Type":"ContainerDied","Data":"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"}
Nov 01 20:44:45.169862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.169806    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn" event={"ID":"d828bb2f-3026-47e9-a96b-53b13430f9bf","Type":"ContainerDied","Data":"8fe3c25b096c920b389d57075678534e54dd613a1e6979b6de4896fc6f59c69e"}
Nov 01 20:44:45.169862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.169834    5066 scope.go:122] "RemoveContainer" containerID="d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"
Nov 01 20:44:45.179918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.179898    5066 scope.go:122] "RemoveContainer" containerID="d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"
Nov 01 20:44:45.180200 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:45.180180    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80\": container with ID starting with d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80 not found: ID does not exist" containerID="d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"
Nov 01 20:44:45.180264 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.180208    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80"} err="failed to get container status \"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80\": rpc error: code = NotFound desc = could not find container \"d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80\": container with ID starting with d1430384b2ff836e0fbfdd69cc4734201ae895f390b202e674948bced4b2bf80 not found: ID does not exist"
Nov 01 20:44:45.280800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.280766    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:45.282245 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.282213    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:44:45.349586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.349553    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:44:45.391845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:45.391814    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8cwvn"]
Nov 01 20:44:46.730978 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:46.730946    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" path="/var/lib/kubelet/pods/d828bb2f-3026-47e9-a96b-53b13430f9bf/volumes"
Nov 01 20:44:49.867704 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:49.867663    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:44:49.868171 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:49.867933    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy" containerID="cri-o://49099708c7abb90e74db90199880722d7052d7f8ee73875c1c4461b6951b4b20" gracePeriod=30
Nov 01 20:44:50.281820 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.281777    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing" containerID="cri-o://61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531" gracePeriod=25
Nov 01 20:44:50.551970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.551943    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:50.692889 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.692857    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\") pod \"8310c81d-3e39-42d9-85db-f1445a1b2112\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") "
Nov 01 20:44:50.693098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.692929    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\") pod \"8310c81d-3e39-42d9-85db-f1445a1b2112\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") "
Nov 01 20:44:50.693098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.692980    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\") pod \"8310c81d-3e39-42d9-85db-f1445a1b2112\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") "
Nov 01 20:44:50.693098 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.693025    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\") pod \"8310c81d-3e39-42d9-85db-f1445a1b2112\" (UID: \"8310c81d-3e39-42d9-85db-f1445a1b2112\") "
Nov 01 20:44:50.693361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.693336    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume" pod "8310c81d-3e39-42d9-85db-f1445a1b2112" (UID: "8310c81d-3e39-42d9-85db-f1445a1b2112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:50.693433 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.693407    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume" pod "8310c81d-3e39-42d9-85db-f1445a1b2112" (UID: "8310c81d-3e39-42d9-85db-f1445a1b2112"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:50.695090 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.695069    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials" pod "8310c81d-3e39-42d9-85db-f1445a1b2112" (UID: "8310c81d-3e39-42d9-85db-f1445a1b2112"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:50.695186 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.695168    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd" pod "8310c81d-3e39-42d9-85db-f1445a1b2112" (UID: "8310c81d-3e39-42d9-85db-f1445a1b2112"). InnerVolumeSpecName "kube-api-access-8clfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:50.795053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.794881    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:50.795053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.794917    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8clfd\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-kube-api-access-8clfd\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:50.795053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.794933    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8310c81d-3e39-42d9-85db-f1445a1b2112-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:50.795053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:50.794946    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8310c81d-3e39-42d9-85db-f1445a1b2112-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:51.208123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.208040    5066 generic.go:356] "Generic (PLEG): container finished" podID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerID="61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531" exitCode=0
Nov 01 20:44:51.208123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.208089    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"
Nov 01 20:44:51.208571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.208124    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" event={"ID":"8310c81d-3e39-42d9-85db-f1445a1b2112","Type":"ContainerDied","Data":"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"}
Nov 01 20:44:51.208571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.208176    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6" event={"ID":"8310c81d-3e39-42d9-85db-f1445a1b2112","Type":"ContainerDied","Data":"bb8f2dd9a80894c4d216aa94cbe19c66e10ec3777b5a644a8fae0d8780e230c2"}
Nov 01 20:44:51.208571 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.208203    5066 scope.go:122] "RemoveContainer" containerID="61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"
Nov 01 20:44:51.218364 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.218341    5066 scope.go:122] "RemoveContainer" containerID="61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"
Nov 01 20:44:51.218643 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:51.218624    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531\": container with ID starting with 61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531 not found: ID does not exist" containerID="61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"
Nov 01 20:44:51.218720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.218650    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531"} err="failed to get container status \"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531\": rpc error: code = NotFound desc = could not find container \"61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531\": container with ID starting with 61dae78b4ebf48feb2653b082d7b5a21aa5558663e1d686a08c36ae66b0a8531 not found: ID does not exist"
Nov 01 20:44:51.370297 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.370262    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:51.419872 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.419838    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-84d4qxrx6"]
Nov 01 20:44:51.422297 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422270    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:44:51.422767 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422753    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy"
Nov 01 20:44:51.422816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422769    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy"
Nov 01 20:44:51.422816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422786    5066 cpu_manager.go:414] "containerMap: removing container" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerName="payload-processing"
Nov 01 20:44:51.422816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422790    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerName="payload-processing"
Nov 01 20:44:51.422816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422802    5066 cpu_manager.go:414] "containerMap: removing container" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing"
Nov 01 20:44:51.422816 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422807    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing"
Nov 01 20:44:51.423025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422879    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" containerName="payload-processing"
Nov 01 20:44:51.423025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422891    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d828bb2f-3026-47e9-a96b-53b13430f9bf" containerName="istio-proxy"
Nov 01 20:44:51.423025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.422898    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="162afbbc-c951-45a3-a6fb-441ff0c6e05e" containerName="payload-processing"
Nov 01 20:44:51.427473 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.427455    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.463265 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.463190    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:44:51.479866 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.479837    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 20:44:51.480088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.479906    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qz59c\""
Nov 01 20:44:51.480088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.479921    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 20:44:51.480088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.479974    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 20:44:51.480088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.479980    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 20:44:51.480088 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.480045    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ca90c301433\""
Nov 01 20:44:51.502122 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.502097    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.502226 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.502153    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.502381 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.502359    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.502440 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.502402    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfhm\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.602942 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.602911    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.603131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.603039    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.603131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.603064    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfhm\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.603131 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.603109    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.603757 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.603723    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.603960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.603938    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.606116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.606095    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.631947 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:44:51.631909    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:44:51.659034 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.659008    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfhm\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\") pod \"payload-processing-external-model-65b97f6f5b-2fpk5\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") " pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:51.740795 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:51.740754    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:52.119673 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.119637    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:44:52.120822 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:44:52.120796    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e777476_8ae1_4258_b85c_948bb4f2c494.slice/crio-c86f2d10ecbc51b24b7b95cd71de6d35161ae9ecbb64457eb481c6352a9d5db4 WatchSource:0}: Error finding container c86f2d10ecbc51b24b7b95cd71de6d35161ae9ecbb64457eb481c6352a9d5db4: Status 404 returned error can't find the container with id c86f2d10ecbc51b24b7b95cd71de6d35161ae9ecbb64457eb481c6352a9d5db4
Nov 01 20:44:52.216130 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.216102    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" event={"ID":"1e777476-8ae1-4258-b85c-948bb4f2c494","Type":"ContainerStarted","Data":"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"}
Nov 01 20:44:52.216461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.216150    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" event={"ID":"1e777476-8ae1-4258-b85c-948bb4f2c494","Type":"ContainerStarted","Data":"c86f2d10ecbc51b24b7b95cd71de6d35161ae9ecbb64457eb481c6352a9d5db4"}
Nov 01 20:44:52.216461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.216176    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:44:52.330119 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.330069    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" podStartSLOduration=1.330054082 podStartE2EDuration="1.330054082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:52.328599842 +0000 UTC m=+2545.713497433" watchObservedRunningTime="2026-11-01 20:44:52.330054082 +0000 UTC m=+2545.714951708"
Nov 01 20:44:52.730114 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:52.730081    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8310c81d-3e39-42d9-85db-f1445a1b2112" path="/var/lib/kubelet/pods/8310c81d-3e39-42d9-85db-f1445a1b2112/volumes"
Nov 01 20:44:55.658106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.657342    5066 generic.go:356] "Generic (PLEG): container finished" podID="1ddd40f0-bd75-4c34-a188-203fde318804" containerID="49099708c7abb90e74db90199880722d7052d7f8ee73875c1c4461b6951b4b20" exitCode=0
Nov 01 20:44:55.658106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.657429    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" event={"ID":"1ddd40f0-bd75-4c34-a188-203fde318804","Type":"ContainerDied","Data":"49099708c7abb90e74db90199880722d7052d7f8ee73875c1c4461b6951b4b20"}
Nov 01 20:44:55.690565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.675737    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:44:55.790389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.790265    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.791568 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.791162    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.791881 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.791852    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.792321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.792303    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.792517 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.792480    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.793058 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.793040    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.793228 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.793210    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.793422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.793390    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.793654 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.793635    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\") pod \"1ddd40f0-bd75-4c34-a188-203fde318804\" (UID: \"1ddd40f0-bd75-4c34-a188-203fde318804\") "
Nov 01 20:44:55.795256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.792237    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:55.795256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.792823    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:55.795256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.794813    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:55.795256 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.795010    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:55.799833 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.799802    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.799990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.799972    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.800128 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.800113    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.800247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.800233    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddd40f0-bd75-4c34-a188-203fde318804-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.804659 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.804614    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:44:55.804914 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.804894    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:55.805111 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.805078    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:55.815524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.812791    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:44:55.831615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.828679    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z" pod "1ddd40f0-bd75-4c34-a188-203fde318804" (UID: "1ddd40f0-bd75-4c34-a188-203fde318804"). InnerVolumeSpecName "kube-api-access-vsx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:55.913899 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.913838    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsx7z\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-kube-api-access-vsx7z\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.927544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.913875    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.927544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.914133    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddd40f0-bd75-4c34-a188-203fde318804-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.927544 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.914149    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddd40f0-bd75-4c34-a188-203fde318804-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:55.928516 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:55.914162    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddd40f0-bd75-4c34-a188-203fde318804-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:44:56.664215 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:56.664187    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"
Nov 01 20:44:56.664740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:56.664192    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq" event={"ID":"1ddd40f0-bd75-4c34-a188-203fde318804","Type":"ContainerDied","Data":"71647fd84619f64749d91ebc1947a39d5a35ebe8ec68358a51069c3a4323ff7a"}
Nov 01 20:44:56.664740 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:56.664326    5066 scope.go:122] "RemoveContainer" containerID="49099708c7abb90e74db90199880722d7052d7f8ee73875c1c4461b6951b4b20"
Nov 01 20:44:56.828187 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:56.828152    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:44:56.877987 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:56.877953    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-q2hlq"]
Nov 01 20:44:58.730726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:44:58.730676    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" path="/var/lib/kubelet/pods/1ddd40f0-bd75-4c34-a188-203fde318804/volumes"
Nov 01 20:45:00.159995 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.159937    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:00.160687 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.160670    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy"
Nov 01 20:45:00.160750 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.160691    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy"
Nov 01 20:45:00.160855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.160844    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddd40f0-bd75-4c34-a188-203fde318804" containerName="istio-proxy"
Nov 01 20:45:00.165946 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.165924    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:00.166053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.166032    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.170144 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.170117    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.217222 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.217192    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\""
Nov 01 20:45:00.270465 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.270416    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:00.273678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.273644    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:00.277781 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.277740    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.277957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.277857    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.277957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.277936    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhcc\" (UniqueName: \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.278093 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.277975    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h4m\" (UniqueName: \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.379509 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.379463    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.379729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.379537    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhcc\" (UniqueName: \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.379729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.379573    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h4m\" (UniqueName: \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.379729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.379719    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.380285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.380262    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.380366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.380348    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.437362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.437292    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhcc\" (UniqueName: \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.437539 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.437446    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h4m\" (UniqueName: \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.444201 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.444174    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:45:00.448123 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.448105    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.480753 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.480717    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:00.481080 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.481061    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.481196 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.481181    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.481316 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.481288    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgmd\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.481426 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.481410    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.487862 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.487837    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501309    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cmp-legacy-1bfe8a-provider-cred\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501330    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501426    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501480    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501628    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.501838    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-bdx45\""
Nov 01 20:45:00.506064 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.505772    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:45:00.583176 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.583129    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgmd\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.583328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.583235    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.583328 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.583293    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.583825 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.583409    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.584258 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.584233    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.584348 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.584317    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.585896 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.585879    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.639529 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.639484    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgmd\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.768288 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.768251    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:00.876112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.876041    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:00.878126 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:45:00.878098    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49aea71_df8e_4c54_8c1a_af4264551596.slice/crio-0955629a67472a62b45eda8d18ecaacb90e118d2e4b3b59de9e381778e7c5d6c WatchSource:0}: Error finding container 0955629a67472a62b45eda8d18ecaacb90e118d2e4b3b59de9e381778e7c5d6c: Status 404 returned error can't find the container with id 0955629a67472a62b45eda8d18ecaacb90e118d2e4b3b59de9e381778e7c5d6c
Nov 01 20:45:00.926071 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:00.926044    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:00.927997 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:45:00.927965    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27ad59_b007_4583_a56f_d10464552a71.slice/crio-a6c1a5e633edd265be244f8724ea4eafd9fb705f57c80881d51ec7078954b5ba WatchSource:0}: Error finding container a6c1a5e633edd265be244f8724ea4eafd9fb705f57c80881d51ec7078954b5ba: Status 404 returned error can't find the container with id a6c1a5e633edd265be244f8724ea4eafd9fb705f57c80881d51ec7078954b5ba
Nov 01 20:45:01.157373 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.157335    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:45:01.159035 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:45:01.159009    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e9b155_ba54_4d9e_8e8a_129bc754d8a8.slice/crio-a5e09ac96c28a2e52d658126a419bbbc04cff0463e6daa744e0eaf4f949114af WatchSource:0}: Error finding container a5e09ac96c28a2e52d658126a419bbbc04cff0463e6daa744e0eaf4f949114af: Status 404 returned error can't find the container with id a5e09ac96c28a2e52d658126a419bbbc04cff0463e6daa744e0eaf4f949114af
Nov 01 20:45:01.696437 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.696396    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerStarted","Data":"9d2b555a15078eb49f74ea4cd84cead80ed827c58b660f13e1d18f4a3de36838"}
Nov 01 20:45:01.696923 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.696527    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerStarted","Data":"0955629a67472a62b45eda8d18ecaacb90e118d2e4b3b59de9e381778e7c5d6c"}
Nov 01 20:45:01.698182 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.698158    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerStarted","Data":"815d1823623b3711f7a869a0ba9fdd26beb86ad5e7cb7520a161039acf435469"}
Nov 01 20:45:01.698301 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.698200    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerStarted","Data":"a6c1a5e633edd265be244f8724ea4eafd9fb705f57c80881d51ec7078954b5ba"}
Nov 01 20:45:01.699779 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.699757    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" event={"ID":"87e9b155-ba54-4d9e-8e8a-129bc754d8a8","Type":"ContainerStarted","Data":"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"}
Nov 01 20:45:01.699880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.699792    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" event={"ID":"87e9b155-ba54-4d9e-8e8a-129bc754d8a8","Type":"ContainerStarted","Data":"a5e09ac96c28a2e52d658126a419bbbc04cff0463e6daa744e0eaf4f949114af"}
Nov 01 20:45:01.699880 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.699869    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:01.809140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.809080    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" podStartSLOduration=1.8090658880000001 podStartE2EDuration="1.809065888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:01.807296506 +0000 UTC m=+2555.192194095" watchObservedRunningTime="2026-11-01 20:45:01.809065888 +0000 UTC m=+2555.193963477"
Nov 01 20:45:01.920047 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:01.919983    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" podStartSLOduration=1.919968269 podStartE2EDuration="1.919968269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:01.918045666 +0000 UTC m=+2555.302943290" watchObservedRunningTime="2026-11-01 20:45:01.919968269 +0000 UTC m=+2555.304865858"
Nov 01 20:45:02.039107 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.039045    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" podStartSLOduration=2.039027777 podStartE2EDuration="2.039027777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:45:02.038070686 +0000 UTC m=+2555.422968275" watchObservedRunningTime="2026-11-01 20:45:02.039027777 +0000 UTC m=+2555.423925371"
Nov 01 20:45:02.707489 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.707451    5066 generic.go:356] "Generic (PLEG): container finished" podID="c49aea71-df8e-4c54-8c1a-af4264551596" containerID="9d2b555a15078eb49f74ea4cd84cead80ed827c58b660f13e1d18f4a3de36838" exitCode=60
Nov 01 20:45:02.707981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.707546    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerDied","Data":"9d2b555a15078eb49f74ea4cd84cead80ed827c58b660f13e1d18f4a3de36838"}
Nov 01 20:45:02.707981 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.707793    5066 scope.go:122] "RemoveContainer" containerID="9d2b555a15078eb49f74ea4cd84cead80ed827c58b660f13e1d18f4a3de36838"
Nov 01 20:45:02.709257 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.709210    5066 generic.go:356] "Generic (PLEG): container finished" podID="4f27ad59-b007-4583-a56f-d10464552a71" containerID="815d1823623b3711f7a869a0ba9fdd26beb86ad5e7cb7520a161039acf435469" exitCode=60
Nov 01 20:45:02.709323 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.709304    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerDied","Data":"815d1823623b3711f7a869a0ba9fdd26beb86ad5e7cb7520a161039acf435469"}
Nov 01 20:45:02.709477 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:02.709461    5066 scope.go:122] "RemoveContainer" containerID="815d1823623b3711f7a869a0ba9fdd26beb86ad5e7cb7520a161039acf435469"
Nov 01 20:45:03.225110 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.225070    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:03.717172 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.717136    5066 generic.go:356] "Generic (PLEG): container finished" podID="c49aea71-df8e-4c54-8c1a-af4264551596" containerID="2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5" exitCode=60
Nov 01 20:45:03.717617 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.717211    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerDied","Data":"2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5"}
Nov 01 20:45:03.717617 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.717270    5066 scope.go:122] "RemoveContainer" containerID="9d2b555a15078eb49f74ea4cd84cead80ed827c58b660f13e1d18f4a3de36838"
Nov 01 20:45:03.717617 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.717525    5066 scope.go:122] "RemoveContainer" containerID="2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5"
Nov 01 20:45:03.717863 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:03.717838    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl_odh-ai-gateway-infra(c49aea71-df8e-4c54-8c1a-af4264551596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" podUID="c49aea71-df8e-4c54-8c1a-af4264551596"
Nov 01 20:45:03.719104 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.719079    5066 generic.go:356] "Generic (PLEG): container finished" podID="4f27ad59-b007-4583-a56f-d10464552a71" containerID="07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7" exitCode=60
Nov 01 20:45:03.719210 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.719164    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerDied","Data":"07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7"}
Nov 01 20:45:03.719402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.719358    5066 scope.go:122] "RemoveContainer" containerID="07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7"
Nov 01 20:45:03.719609 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:03.719591    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg_odh-ai-gateway-infra(4f27ad59-b007-4583-a56f-d10464552a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" podUID="4f27ad59-b007-4583-a56f-d10464552a71"
Nov 01 20:45:03.729029 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:03.729007    5066 scope.go:122] "RemoveContainer" containerID="815d1823623b3711f7a869a0ba9fdd26beb86ad5e7cb7520a161039acf435469"
Nov 01 20:45:04.065478 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:04.065393    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:45:04.066887 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:04.066854    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:04.727073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:04.727044    5066 scope.go:122] "RemoveContainer" containerID="2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5"
Nov 01 20:45:04.727510 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:04.727297    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl_odh-ai-gateway-infra(c49aea71-df8e-4c54-8c1a-af4264551596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" podUID="c49aea71-df8e-4c54-8c1a-af4264551596"
Nov 01 20:45:04.728727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:04.728708    5066 scope.go:122] "RemoveContainer" containerID="07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7"
Nov 01 20:45:04.728902 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:04.728886    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg_odh-ai-gateway-infra(4f27ad59-b007-4583-a56f-d10464552a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" podUID="4f27ad59-b007-4583-a56f-d10464552a71"
Nov 01 20:45:09.065855 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.065645    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing" containerID="cri-o://cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88" gracePeriod=25
Nov 01 20:45:09.356240 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.356211    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:45:09.489545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.487112    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\") pod \"1e777476-8ae1-4258-b85c-948bb4f2c494\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") "
Nov 01 20:45:09.489545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.487272    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\") pod \"1e777476-8ae1-4258-b85c-948bb4f2c494\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") "
Nov 01 20:45:09.489545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.487356    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\") pod \"1e777476-8ae1-4258-b85c-948bb4f2c494\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") "
Nov 01 20:45:09.489545 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.487388    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\") pod \"1e777476-8ae1-4258-b85c-948bb4f2c494\" (UID: \"1e777476-8ae1-4258-b85c-948bb4f2c494\") "
Nov 01 20:45:09.490404 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.490367    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume" pod "1e777476-8ae1-4258-b85c-948bb4f2c494" (UID: "1e777476-8ae1-4258-b85c-948bb4f2c494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:09.490783 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.490757    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume" pod "1e777476-8ae1-4258-b85c-948bb4f2c494" (UID: "1e777476-8ae1-4258-b85c-948bb4f2c494"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:09.504112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.504051    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials" pod "1e777476-8ae1-4258-b85c-948bb4f2c494" (UID: "1e777476-8ae1-4258-b85c-948bb4f2c494"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:09.504112 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.504064    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm" pod "1e777476-8ae1-4258-b85c-948bb4f2c494" (UID: "1e777476-8ae1-4258-b85c-948bb4f2c494"). InnerVolumeSpecName "kube-api-access-4lfhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:09.588754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.588664    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.588754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.588698    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lfhm\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-kube-api-access-4lfhm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.588754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.588712    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e777476-8ae1-4258-b85c-948bb4f2c494-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.588754 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.588725    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e777476-8ae1-4258-b85c-948bb4f2c494-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.761507 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.761449    5066 generic.go:356] "Generic (PLEG): container finished" podID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerID="cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88" exitCode=0
Nov 01 20:45:09.761678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.761535    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"
Nov 01 20:45:09.761678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.761545    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" event={"ID":"1e777476-8ae1-4258-b85c-948bb4f2c494","Type":"ContainerDied","Data":"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"}
Nov 01 20:45:09.761678 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.761628    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5" event={"ID":"1e777476-8ae1-4258-b85c-948bb4f2c494","Type":"ContainerDied","Data":"c86f2d10ecbc51b24b7b95cd71de6d35161ae9ecbb64457eb481c6352a9d5db4"}
Nov 01 20:45:09.761891 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.761681    5066 scope.go:122] "RemoveContainer" containerID="cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"
Nov 01 20:45:09.774615 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.774585    5066 scope.go:122] "RemoveContainer" containerID="cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"
Nov 01 20:45:09.774896 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:09.774864    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88\": container with ID starting with cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88 not found: ID does not exist" containerID="cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"
Nov 01 20:45:09.774972 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.774907    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88"} err="failed to get container status \"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88\": rpc error: code = NotFound desc = could not find container \"cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88\": container with ID starting with cc5e7a1769748fbdb294b571ce5855e35f26327cb63891706f4c1f514b8aba88 not found: ID does not exist"
Nov 01 20:45:09.936574 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.935614    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:45:09.941521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:09.940488    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b97f6f5b-2fpk5"]
Nov 01 20:45:10.788733 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:10.788692    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" path="/var/lib/kubelet/pods/1e777476-8ae1-4258-b85c-948bb4f2c494/volumes"
Nov 01 20:45:10.799014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:10.798984    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:45:12.713337 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:12.713308    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:45:14.724453 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:14.724422    5066 scope.go:122] "RemoveContainer" containerID="07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7"
Nov 01 20:45:14.724846 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:14.724528    5066 scope.go:122] "RemoveContainer" containerID="2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5"
Nov 01 20:45:15.800535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.800473    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing" containerID="cri-o://bae77dc350202b5eaf880eb534d7b846f2fa211bdc4fd2d6f041fc34847e493b" gracePeriod=25
Nov 01 20:45:15.874151 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.874118    5066 generic.go:356] "Generic (PLEG): container finished" podID="c49aea71-df8e-4c54-8c1a-af4264551596" containerID="7b01cf17f7a8c5996f96ada6635b6c20a45dcade403551dc1be8daeee57874eb" exitCode=60
Nov 01 20:45:15.874320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.874193    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerDied","Data":"7b01cf17f7a8c5996f96ada6635b6c20a45dcade403551dc1be8daeee57874eb"}
Nov 01 20:45:15.874320 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.874281    5066 scope.go:122] "RemoveContainer" containerID="2cc7adc241d91bcf6b660b39194f20a14108fbc399a6ddec6f9a060fdefca0e5"
Nov 01 20:45:15.874774 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.874674    5066 scope.go:122] "RemoveContainer" containerID="7b01cf17f7a8c5996f96ada6635b6c20a45dcade403551dc1be8daeee57874eb"
Nov 01 20:45:15.875183 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:15.875153    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl_odh-ai-gateway-infra(c49aea71-df8e-4c54-8c1a-af4264551596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" podUID="c49aea71-df8e-4c54-8c1a-af4264551596"
Nov 01 20:45:15.876506 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.876469    5066 generic.go:356] "Generic (PLEG): container finished" podID="4f27ad59-b007-4583-a56f-d10464552a71" containerID="cf9d064d2c16831cb0b9b331a64f1c26988ddcfb9dabfd6c7a68109386781c44" exitCode=60
Nov 01 20:45:15.876611 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.876489    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerDied","Data":"cf9d064d2c16831cb0b9b331a64f1c26988ddcfb9dabfd6c7a68109386781c44"}
Nov 01 20:45:15.876845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.876831    5066 scope.go:122] "RemoveContainer" containerID="cf9d064d2c16831cb0b9b331a64f1c26988ddcfb9dabfd6c7a68109386781c44"
Nov 01 20:45:15.877081 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:15.877061    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg_odh-ai-gateway-infra(4f27ad59-b007-4583-a56f-d10464552a71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" podUID="4f27ad59-b007-4583-a56f-d10464552a71"
Nov 01 20:45:15.886441 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:15.886424    5066 scope.go:122] "RemoveContainer" containerID="07fc88e58ad097ad8ed8b1b86135d79306986b7b1ff1ba303d51bb80f59708b7"
Nov 01 20:45:16.317636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:16.317601    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:16.997150 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:16.997118    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:17.097214 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.097038    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:17.145005 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.144915    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\") pod \"4f27ad59-b007-4583-a56f-d10464552a71\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") "
Nov 01 20:45:17.145174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.145103    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\" (UniqueName: \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\") pod \"4f27ad59-b007-4583-a56f-d10464552a71\" (UID: \"4f27ad59-b007-4583-a56f-d10464552a71\") "
Nov 01 20:45:17.145270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.145244    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca" pod "4f27ad59-b007-4583-a56f-d10464552a71" (UID: "4f27ad59-b007-4583-a56f-d10464552a71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:17.145475 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.145458    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27ad59-b007-4583-a56f-d10464552a71-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:17.147302 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.147276    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc" pod "4f27ad59-b007-4583-a56f-d10464552a71" (UID: "4f27ad59-b007-4583-a56f-d10464552a71"). InnerVolumeSpecName "kube-api-access-4dhcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:17.206830 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.206805    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:17.246706 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.246674    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhcc\" (UniqueName: \"kubernetes.io/projected/4f27ad59-b007-4583-a56f-d10464552a71-kube-api-access-4dhcc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:17.347843 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.347807    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\") pod \"c49aea71-df8e-4c54-8c1a-af4264551596\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") "
Nov 01 20:45:17.348040 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.347974    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\" (UniqueName: \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\") pod \"c49aea71-df8e-4c54-8c1a-af4264551596\" (UID: \"c49aea71-df8e-4c54-8c1a-af4264551596\") "
Nov 01 20:45:17.348260 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.348235    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca" pod "c49aea71-df8e-4c54-8c1a-af4264551596" (UID: "c49aea71-df8e-4c54-8c1a-af4264551596"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:17.348389 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.348375    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49aea71-df8e-4c54-8c1a-af4264551596-maas-api-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:17.349989 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.349965    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m" pod "c49aea71-df8e-4c54-8c1a-af4264551596" (UID: "c49aea71-df8e-4c54-8c1a-af4264551596"). InnerVolumeSpecName "kube-api-access-h9h4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:17.381595 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.381555    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:17.449682 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.449601    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h4m\" (UniqueName: \"kubernetes.io/projected/c49aea71-df8e-4c54-8c1a-af4264551596-kube-api-access-h9h4m\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:17.894653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.894623    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"
Nov 01 20:45:17.894653 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.894643    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl" event={"ID":"c49aea71-df8e-4c54-8c1a-af4264551596","Type":"ContainerDied","Data":"0955629a67472a62b45eda8d18ecaacb90e118d2e4b3b59de9e381778e7c5d6c"}
Nov 01 20:45:17.894875 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.894707    5066 scope.go:122] "RemoveContainer" containerID="7b01cf17f7a8c5996f96ada6635b6c20a45dcade403551dc1be8daeee57874eb"
Nov 01 20:45:17.896206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.896184    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"
Nov 01 20:45:17.896206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.896192    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg" event={"ID":"4f27ad59-b007-4583-a56f-d10464552a71","Type":"ContainerDied","Data":"a6c1a5e633edd265be244f8724ea4eafd9fb705f57c80881d51ec7078954b5ba"}
Nov 01 20:45:17.909456 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:17.909437    5066 scope.go:122] "RemoveContainer" containerID="cf9d064d2c16831cb0b9b331a64f1c26988ddcfb9dabfd6c7a68109386781c44"
Nov 01 20:45:18.058941 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.058894    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:18.061437 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.061410    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892765-55dsl"]
Nov 01 20:45:18.168773 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.168693    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:18.172716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.172691    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-df41b8-29892765-cgsgg"]
Nov 01 20:45:18.731813 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.731781    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f27ad59-b007-4583-a56f-d10464552a71" path="/var/lib/kubelet/pods/4f27ad59-b007-4583-a56f-d10464552a71/volumes"
Nov 01 20:45:18.732185 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:18.732170    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" path="/var/lib/kubelet/pods/c49aea71-df8e-4c54-8c1a-af4264551596/volumes"
Nov 01 20:45:21.630260 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:21.630217    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:45:27.381542 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:27.381473    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:36.047202 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.047165    5066 generic.go:356] "Generic (PLEG): container finished" podID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerID="bae77dc350202b5eaf880eb534d7b846f2fa211bdc4fd2d6f041fc34847e493b" exitCode=0
Nov 01 20:45:36.047845 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.047717    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" event={"ID":"b616f7de-b7e5-4f09-ab82-695dbb43e1d3","Type":"ContainerDied","Data":"bae77dc350202b5eaf880eb534d7b846f2fa211bdc4fd2d6f041fc34847e493b"}
Nov 01 20:45:36.068247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.068220    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:45:36.140321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.140284    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\") pod \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") "
Nov 01 20:45:36.140512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.140430    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\" (UniqueName: \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\") pod \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\" (UID: \"b616f7de-b7e5-4f09-ab82-695dbb43e1d3\") "
Nov 01 20:45:36.140746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.140718    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume" pod "b616f7de-b7e5-4f09-ab82-695dbb43e1d3" (UID: "b616f7de-b7e5-4f09-ab82-695dbb43e1d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:36.140867 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.140850    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:36.142636 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.142607    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp" pod "b616f7de-b7e5-4f09-ab82-695dbb43e1d3" (UID: "b616f7de-b7e5-4f09-ab82-695dbb43e1d3"). InnerVolumeSpecName "kube-api-access-xf5tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:36.242436 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:36.242398    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf5tp\" (UniqueName: \"kubernetes.io/projected/b616f7de-b7e5-4f09-ab82-695dbb43e1d3-kube-api-access-xf5tp\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:45:37.054313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:37.054233    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"
Nov 01 20:45:37.054313 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:37.054245    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz" event={"ID":"b616f7de-b7e5-4f09-ab82-695dbb43e1d3","Type":"ContainerDied","Data":"45fafc8d741e35bd179a2b263aa26d066da682de8a4e10b6e91c584f9a08b664"}
Nov 01 20:45:37.054846 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:37.054330    5066 scope.go:122] "RemoveContainer" containerID="bae77dc350202b5eaf880eb534d7b846f2fa211bdc4fd2d6f041fc34847e493b"
Nov 01 20:45:37.216512 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:37.216464    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:45:37.265524 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:37.265481    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-df41b8-5d449c59f5-v6qjz"]
Nov 01 20:45:38.730716 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:38.730679    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" path="/var/lib/kubelet/pods/b616f7de-b7e5-4f09-ab82-695dbb43e1d3/volumes"
Nov 01 20:45:51.628722 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:45:51.628675    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:45:55.685727 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:45:55.685692    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:46:00.686966 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:00.686923    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" podUID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" containerName="payload-processing" containerID="cri-o://3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9" gracePeriod=25
Nov 01 20:46:00.940832 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:00.940770    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:46:01.118812 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.118777    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\") pod \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") "
Nov 01 20:46:01.119008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.118860    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\") pod \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") "
Nov 01 20:46:01.119008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.118880    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\") pod \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") "
Nov 01 20:46:01.119008 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.118915    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\") pod \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\" (UID: \"87e9b155-ba54-4d9e-8e8a-129bc754d8a8\") "
Nov 01 20:46:01.119247 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.119222    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume" pod "87e9b155-ba54-4d9e-8e8a-129bc754d8a8" (UID: "87e9b155-ba54-4d9e-8e8a-129bc754d8a8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:46:01.119356 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.119329    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume" pod "87e9b155-ba54-4d9e-8e8a-129bc754d8a8" (UID: "87e9b155-ba54-4d9e-8e8a-129bc754d8a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:46:01.121011 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.120987    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials" pod "87e9b155-ba54-4d9e-8e8a-129bc754d8a8" (UID: "87e9b155-ba54-4d9e-8e8a-129bc754d8a8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:01.121106 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.121090    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd" pod "87e9b155-ba54-4d9e-8e8a-129bc754d8a8" (UID: "87e9b155-ba54-4d9e-8e8a-129bc754d8a8"). InnerVolumeSpecName "kube-api-access-7qgmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:01.216115 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.216083    5066 generic.go:356] "Generic (PLEG): container finished" podID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" containerID="3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9" exitCode=0
Nov 01 20:46:01.216319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.216128    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"
Nov 01 20:46:01.216319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.216149    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" event={"ID":"87e9b155-ba54-4d9e-8e8a-129bc754d8a8","Type":"ContainerDied","Data":"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"}
Nov 01 20:46:01.216319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.216187    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d" event={"ID":"87e9b155-ba54-4d9e-8e8a-129bc754d8a8","Type":"ContainerDied","Data":"a5e09ac96c28a2e52d658126a419bbbc04cff0463e6daa744e0eaf4f949114af"}
Nov 01 20:46:01.216319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.216210    5066 scope.go:122] "RemoveContainer" containerID="3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"
Nov 01 20:46:01.219764 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.219744    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:46:01.219852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.219766    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgmd\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-kube-api-access-7qgmd\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:46:01.219852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.219777    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:46:01.219852 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.219786    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e9b155-ba54-4d9e-8e8a-129bc754d8a8-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:46:01.226291 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.226273    5066 scope.go:122] "RemoveContainer" containerID="3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"
Nov 01 20:46:01.226573 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:01.226551    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9\": container with ID starting with 3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9 not found: ID does not exist" containerID="3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"
Nov 01 20:46:01.226632 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.226588    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9"} err="failed to get container status \"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9\": rpc error: code = NotFound desc = could not find container \"3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9\": container with ID starting with 3f40728a3463dc5e633a650877c76202f45d483c906191314a34d18b066368a9 not found: ID does not exist"
Nov 01 20:46:01.379531 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.379480    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:46:01.382768 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:01.382740    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5db4h494d"]
Nov 01 20:46:02.321546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.321475    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"]
Nov 01 20:46:02.322174 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322152    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322176    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322194    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322200    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322222    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322230    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322248    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322254    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing"
Nov 01 20:46:02.322272 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322270    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322276    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322288    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322294    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322313    5066 cpu_manager.go:414] "containerMap: removing container" podUID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322321    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322333    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322340    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322473    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322487    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322511    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b616f7de-b7e5-4f09-ab82-695dbb43e1d3" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322525    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322535    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322547    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e777476-8ae1-4258-b85c-948bb4f2c494" containerName="payload-processing"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322557    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.322718 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.322567    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49aea71-df8e-4c54-8c1a-af4264551596" containerName="cleanup"
Nov 01 20:46:02.327319 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.327300    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.382416 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.382382    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"]
Nov 01 20:46:02.429035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429001    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-71951cbb86be\""
Nov 01 20:46:02.429035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429030    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 20:46:02.429035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429015    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 20:46:02.429035 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429012    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 20:46:02.429338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429072    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\""
Nov 01 20:46:02.429338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.429107    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 20:46:02.432711 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.432690    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.432794 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.432721    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z2g\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.432868 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.432851    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.432910 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.432887    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.534333 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.534295    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.534333 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.534333    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z2g\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.534575 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:02.534447    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:02.534575 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:02.534467    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:02.534575 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.534525    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.534575 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:02.534552    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:03.034537281 +0000 UTC m=+2616.419434854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:02.534715 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.534589    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.535304 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.535279    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.535339 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.535327    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.591454 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.591390    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z2g\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:02.730828 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:02.730795    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e9b155-ba54-4d9e-8e8a-129bc754d8a8" path="/var/lib/kubelet/pods/87e9b155-ba54-4d9e-8e8a-129bc754d8a8/volumes"
Nov 01 20:46:03.039246 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:03.039209    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:03.039443 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:03.039381    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:03.039443 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:03.039409    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:03.039567 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:03.039476    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:04.039457188 +0000 UTC m=+2617.424354770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:04.048729 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.048691    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:04.049217 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:04.048861    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:04.049217 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:04.048881    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:04.049217 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:04.048949    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:06.048934637 +0000 UTC m=+2619.433832217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:04.167918 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.167881    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:46:04.168366 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.168353    5066 cpu_manager.go:414] "containerMap: removing container" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:04.168432 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.168367    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:04.168473 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.168463    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f27ad59-b007-4583-a56f-d10464552a71" containerName="cleanup"
Nov 01 20:46:04.172897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.172873    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.274165 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.274133    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\""
Nov 01 20:46:04.276724 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.276701    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:46:04.352362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.352267    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.352362 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.352324    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79q2\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.352618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.352443    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.352618 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.352592    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.453179 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.453134    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79q2\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.453385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.453198    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.453385 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.453287    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.453532 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.453380    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.454042 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.454012    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.454154 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.454120    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.455658 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.455635    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.509347 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.509314    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79q2\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:04.788422 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:04.788377    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:05.167043 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:05.167012    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:46:05.168540 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:46:05.168487    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789da027_7d9d_4ced_8165_ca18764e53f1.slice/crio-09ca19e3c16b1c62b3eb80be4373452fd9cf657fcbf6f723acb8e0617bb0df5b WatchSource:0}: Error finding container 09ca19e3c16b1c62b3eb80be4373452fd9cf657fcbf6f723acb8e0617bb0df5b: Status 404 returned error can't find the container with id 09ca19e3c16b1c62b3eb80be4373452fd9cf657fcbf6f723acb8e0617bb0df5b
Nov 01 20:46:05.242120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:05.242089    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" event={"ID":"789da027-7d9d-4ced-8165-ca18764e53f1","Type":"ContainerStarted","Data":"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"}
Nov 01 20:46:05.242296 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:05.242153    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" event={"ID":"789da027-7d9d-4ced-8165-ca18764e53f1","Type":"ContainerStarted","Data":"09ca19e3c16b1c62b3eb80be4373452fd9cf657fcbf6f723acb8e0617bb0df5b"}
Nov 01 20:46:05.242374 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:05.242292    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:46:05.355402 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:05.355340    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podStartSLOduration=1.355325229 podStartE2EDuration="1.355325229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:05.354712376 +0000 UTC m=+2618.739609964" watchObservedRunningTime="2026-11-01 20:46:05.355325229 +0000 UTC m=+2618.740222849"
Nov 01 20:46:06.072169 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:06.072126    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:06.072362 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:06.072273    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:06.072362 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:06.072293    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:06.072362 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:06.072346    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:10.072331092 +0000 UTC m=+2623.457228680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:10.114394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:10.114355    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:10.114797 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:10.114539    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:10.114797 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:10.114562    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:10.114797 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:10.114631    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:18.114608702 +0000 UTC m=+2631.499506284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:16.254668 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:16.254630    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:46:18.196609 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:18.196573    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:18.196996 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:18.196722    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:18.196996 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:18.196742    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:18.196996 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:18.196790    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:46:34.196776423 +0000 UTC m=+2647.581673995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:21.629181 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:21.629138    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:46:26.253028 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:26.252984    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:46:34.259712 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:34.259671    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:46:34.260233 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:34.259822    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:34.260233 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:34.259843    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:34.260233 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:34.259897    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:47:06.259882019 +0000 UTC m=+2679.644779587 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:46:36.252731 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:36.252691    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:46:46.253113 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:46.253031    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:46:51.628558 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:46:51.628518    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:46:56.252513 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:46:56.252462    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:06.252384 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:06.252342    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:06.277706 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:06.277671    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:47:06.277875 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:47:06.277810    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:47:06.277875 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:47:06.277828    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:47:06.277875 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:47:06.277872    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:48:10.277859211 +0000 UTC m=+2743.662756783 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:47:07.725121 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:07.725079    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:17.725289 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:17.725250    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:21.628296 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:47:21.628256    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:47:27.217951 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:27.217925    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:47:27.261120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:27.261097    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:47:27.726145 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:27.726107    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:37.725786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:37.725745    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:47.725683 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:47.725644    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:47:51.627840 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:47:51.627803    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:47:57.725664 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:47:57.725627    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:05.340390 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:05.340353    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" podUID="712e7bd9-f740-4566-bffa-e6de313e1426"
Nov 01 20:48:05.919911 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:05.919877    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:48:07.725476 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:07.725435    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:08.725383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:08.725345    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:10.319073 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:10.319026    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:48:10.319459 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:10.319171    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-71951cbb86be: secret "agc-cred-71951cbb86be" not found
Nov 01 20:48:10.319459 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:10.319188    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh: secret "agc-cred-71951cbb86be" not found
Nov 01 20:48:10.319459 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:10.319239    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials podName:712e7bd9-f740-4566-bffa-e6de313e1426 nodeName:}" failed. No retries permitted until 2026-11-01 20:50:12.319226173 +0000 UTC m=+2865.704123754 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426") : secret "agc-cred-71951cbb86be" not found
Nov 01 20:48:18.725423 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:18.725384    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:21.628160 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:21.628122    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:48:28.725960 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:28.725925    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:33.722945 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:33.722906    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"]
Nov 01 20:48:33.723327 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:33.723297    5066 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh" podUID="712e7bd9-f740-4566-bffa-e6de313e1426"
Nov 01 20:48:34.071976 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.071898    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:48:34.078858 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.078836    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:48:34.254261 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.254227    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\") pod \"712e7bd9-f740-4566-bffa-e6de313e1426\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") "
Nov 01 20:48:34.254443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.254288    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\") pod \"712e7bd9-f740-4566-bffa-e6de313e1426\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") "
Nov 01 20:48:34.254443 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.254383    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\") pod \"712e7bd9-f740-4566-bffa-e6de313e1426\" (UID: \"712e7bd9-f740-4566-bffa-e6de313e1426\") "
Nov 01 20:48:34.254726 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.254699    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume" pod "712e7bd9-f740-4566-bffa-e6de313e1426" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:48:34.254856 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.254837    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume" pod "712e7bd9-f740-4566-bffa-e6de313e1426" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:48:34.256456 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.256437    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g" pod "712e7bd9-f740-4566-bffa-e6de313e1426" (UID: "712e7bd9-f740-4566-bffa-e6de313e1426"). InnerVolumeSpecName "kube-api-access-x8z2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:48:34.355719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.355652    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:34.355719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.355679    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8z2g\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-kube-api-access-x8z2g\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:34.355719 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:34.355689    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/712e7bd9-f740-4566-bffa-e6de313e1426-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:35.077521 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:35.077426    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"
Nov 01 20:48:35.301361 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:35.301325    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"]
Nov 01 20:48:35.306786 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:35.306757    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-79d4vbthh"]
Nov 01 20:48:35.466996 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:35.466962    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/712e7bd9-f740-4566-bffa-e6de313e1426-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:36.730762 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:36.730719    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712e7bd9-f740-4566-bffa-e6de313e1426" path="/var/lib/kubelet/pods/712e7bd9-f740-4566-bffa-e6de313e1426/volumes"
Nov 01 20:48:38.725510 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:38.725463    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:46.173383 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:46.173347    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:48:46.174787 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:46.174750    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:48:51.173746 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.173689    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing" containerID="cri-o://b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9" gracePeriod=25
Nov 01 20:48:51.431604 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.431545    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:48:51.523349 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523317    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\") pod \"789da027-7d9d-4ced-8165-ca18764e53f1\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") "
Nov 01 20:48:51.523550 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523358    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\") pod \"789da027-7d9d-4ced-8165-ca18764e53f1\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") "
Nov 01 20:48:51.523550 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523392    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\") pod \"789da027-7d9d-4ced-8165-ca18764e53f1\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") "
Nov 01 20:48:51.523687 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523563    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\") pod \"789da027-7d9d-4ced-8165-ca18764e53f1\" (UID: \"789da027-7d9d-4ced-8165-ca18764e53f1\") "
Nov 01 20:48:51.523744 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523692    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume" pod "789da027-7d9d-4ced-8165-ca18764e53f1" (UID: "789da027-7d9d-4ced-8165-ca18764e53f1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:48:51.523946 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523924    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume" pod "789da027-7d9d-4ced-8165-ca18764e53f1" (UID: "789da027-7d9d-4ced-8165-ca18764e53f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:48:51.524019 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.523942    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:51.525619 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.525596    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2" pod "789da027-7d9d-4ced-8165-ca18764e53f1" (UID: "789da027-7d9d-4ced-8165-ca18764e53f1"). InnerVolumeSpecName "kube-api-access-v79q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:48:51.525720 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.525631    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials" pod "789da027-7d9d-4ced-8165-ca18764e53f1" (UID: "789da027-7d9d-4ced-8165-ca18764e53f1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:48:51.624800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.624765    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789da027-7d9d-4ced-8165-ca18764e53f1-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:51.624800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.624795    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:51.624800 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:51.624805    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79q2\" (UniqueName: \"kubernetes.io/projected/789da027-7d9d-4ced-8165-ca18764e53f1-kube-api-access-v79q2\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:48:51.629297 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:51.629266    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:48:52.181660 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.181616    5066 generic.go:356] "Generic (PLEG): container finished" podID="789da027-7d9d-4ced-8165-ca18764e53f1" containerID="b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9" exitCode=0
Nov 01 20:48:52.182116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.181680    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"
Nov 01 20:48:52.182116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.181703    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" event={"ID":"789da027-7d9d-4ced-8165-ca18764e53f1","Type":"ContainerDied","Data":"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"}
Nov 01 20:48:52.182116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.181804    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8" event={"ID":"789da027-7d9d-4ced-8165-ca18764e53f1","Type":"ContainerDied","Data":"09ca19e3c16b1c62b3eb80be4373452fd9cf657fcbf6f723acb8e0617bb0df5b"}
Nov 01 20:48:52.182116 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.181844    5066 scope.go:122] "RemoveContainer" containerID="b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"
Nov 01 20:48:52.191619 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.191603    5066 scope.go:122] "RemoveContainer" containerID="b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"
Nov 01 20:48:52.191868 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:48:52.191848    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9\": container with ID starting with b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9 not found: ID does not exist" containerID="b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"
Nov 01 20:48:52.191948 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.191879    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9"} err="failed to get container status \"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9\": rpc error: code = NotFound desc = could not find container \"b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9\": container with ID starting with b7fbf3a33c3e937d1cf5723bd2e05d852b25d1dc56b318d8f835ac0a852750a9 not found: ID does not exist"
Nov 01 20:48:52.344586 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.344559    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:48:52.393799 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.393766    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c2mcm8"]
Nov 01 20:48:52.729904 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:48:52.729874    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" path="/var/lib/kubelet/pods/789da027-7d9d-4ced-8165-ca18764e53f1/volumes"
Nov 01 20:49:21.628423 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:49:21.628385    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:49:51.628549 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:49:51.628512    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:50:21.629440 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:50:21.629403    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:50:51.651175 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:50:51.651121    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:51:21.629132 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:51:21.629050    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:51:24.641666 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.641627    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:51:24.642160 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.641873    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" containerID="cri-o://a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070" gracePeriod=30
Nov 01 20:51:24.692307 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.692275    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:51:24.692565 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.692520    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" containerID="cri-o://1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab" gracePeriod=30
Nov 01 20:51:24.695970 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.695938    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:51:24.696178 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:24.696153    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" containerID="cri-o://26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f" gracePeriod=30
Nov 01 20:51:26.938014 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:26.937975    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-z5gnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:51:26.938394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:26.938029    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:51:28.319142 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:28.319104    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-d65sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:51:28.319535 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:28.319174    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:51:28.353784 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:28.353751    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rszdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:51:28.353928 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:28.353817    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:51:29.962077 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:29.962053    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:51:29.996140 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:29.996103    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:51:29.999463 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:29.999444    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:51:30.033120 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.033092    5066 generic.go:356] "Generic (PLEG): container finished" podID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerID="a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070" exitCode=0
Nov 01 20:51:30.033219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.033136    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"
Nov 01 20:51:30.033219 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.033185    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" event={"ID":"98feb445-a93a-4db0-8130-9b46234b7d4c","Type":"ContainerDied","Data":"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"}
Nov 01 20:51:30.033321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.033263    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv" event={"ID":"98feb445-a93a-4db0-8130-9b46234b7d4c","Type":"ContainerDied","Data":"5c6b9b580a4d304707055f50e297a7a800a0258cd58545a33a33940ab6954024"}
Nov 01 20:51:30.033321 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.033282    5066 scope.go:122] "RemoveContainer" containerID="a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"
Nov 01 20:51:30.034523 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.034502    5066 generic.go:356] "Generic (PLEG): container finished" podID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerID="1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab" exitCode=0
Nov 01 20:51:30.034599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.034534    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" event={"ID":"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35","Type":"ContainerDied","Data":"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"}
Nov 01 20:51:30.034599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.034550    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"
Nov 01 20:51:30.034599 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.034581    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb" event={"ID":"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35","Type":"ContainerDied","Data":"08bba42e584d391a05fbbca274724be0896fdf5267392019b6da3fa4f163f52b"}
Nov 01 20:51:30.035865 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.035847    5066 generic.go:356] "Generic (PLEG): container finished" podID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerID="26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f" exitCode=0
Nov 01 20:51:30.035973 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.035902    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"
Nov 01 20:51:30.035973 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.035926    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" event={"ID":"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb","Type":"ContainerDied","Data":"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"}
Nov 01 20:51:30.036054 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.035975    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd" event={"ID":"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb","Type":"ContainerDied","Data":"1670ba1475338cb9a9c4909f684c8d2eb736b14b5d5a260f38b2fb18b228794c"}
Nov 01 20:51:30.042274 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.042255    5066 scope.go:122] "RemoveContainer" containerID="a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"
Nov 01 20:51:30.042551 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:51:30.042528    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070\": container with ID starting with a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070 not found: ID does not exist" containerID="a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"
Nov 01 20:51:30.042614 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.042561    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070"} err="failed to get container status \"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070\": rpc error: code = NotFound desc = could not find container \"a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070\": container with ID starting with a5ded10f310d233a2f99f2b8675f1fbfb12c461986a341ad9887d12cc3760070 not found: ID does not exist"
Nov 01 20:51:30.042614 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.042582    5066 scope.go:122] "RemoveContainer" containerID="1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"
Nov 01 20:51:30.050876 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.050854    5066 scope.go:122] "RemoveContainer" containerID="1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"
Nov 01 20:51:30.051101 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:51:30.051084    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab\": container with ID starting with 1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab not found: ID does not exist" containerID="1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"
Nov 01 20:51:30.051152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.051108    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab"} err="failed to get container status \"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab\": rpc error: code = NotFound desc = could not find container \"1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab\": container with ID starting with 1de8a67af465524fc4db9be98ccabbc5bc54d24949e89fa763b25937fbb2baab not found: ID does not exist"
Nov 01 20:51:30.051152 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.051122    5066 scope.go:122] "RemoveContainer" containerID="26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"
Nov 01 20:51:30.058990 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.058973    5066 scope.go:122] "RemoveContainer" containerID="26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"
Nov 01 20:51:30.059225 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:51:30.059206    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f\": container with ID starting with 26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f not found: ID does not exist" containerID="26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"
Nov 01 20:51:30.059285 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.059230    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f"} err="failed to get container status \"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f\": rpc error: code = NotFound desc = could not find container \"26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f\": container with ID starting with 26371fb636c0fb4deb8149dba4b578bb1d1dda249425138fc50f80d701efe04f not found: ID does not exist"
Nov 01 20:51:30.075044 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075023    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.075136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075063    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075136 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075109    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075149    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.075244 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075187    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075244    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.075343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075280    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075343 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075310    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075346    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075376    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075416    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075446    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075478    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075517    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.075546 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075527    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075635    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075667    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075696    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075717    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075730    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075729    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075742    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075788    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.075897 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075841    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075934    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075971    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.075983    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076020    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076055    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076058    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076089    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\") pod \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\" (UID: \"7bd379cd-cb94-4dd0-b3a3-2fd36c53df35\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076130    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\") pod \"98feb445-a93a-4db0-8130-9b46234b7d4c\" (UID: \"98feb445-a93a-4db0-8130-9b46234b7d4c\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076162    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\") pod \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\" (UID: \"6a7c06d8-e6eb-4575-a7d7-72457c2cabfb\") "
Nov 01 20:51:30.076355 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076341    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076649    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076667    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98feb445-a93a-4db0-8130-9b46234b7d4c-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076686    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076700    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076715    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076730    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.076850 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076744    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.077209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076946    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.077209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.076955    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.077209 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077152    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077535    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077635    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "kube-api-access-8pstw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077772    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077795    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077927    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:51:30.077957 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.077941    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.078370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.078024    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.078370 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.078162    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.079676 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.079655    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:51:30.079859 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.079833    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.079938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.079833    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.079938 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.079906    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:51:30.080164 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.080141    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:51:30.080252 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.080218    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "kube-api-access-rp7nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.080252 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.080219    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy" pod "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" (UID: "7bd379cd-cb94-4dd0-b3a3-2fd36c53df35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:51:30.080325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.080273    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token" pod "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" (UID: "6a7c06d8-e6eb-4575-a7d7-72457c2cabfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.080325 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.080311    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb" pod "98feb445-a93a-4db0-8130-9b46234b7d4c" (UID: "98feb445-a93a-4db0-8130-9b46234b7d4c"). InnerVolumeSpecName "kube-api-access-xqwxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:51:30.177728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177697    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.177728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177721    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.177728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177730    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.177728 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177738    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177747    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177755    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177763    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177771    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177781    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177789    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwxb\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-kube-api-access-xqwxb\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177799    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177807    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177816    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177824    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177832    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98feb445-a93a-4db0-8130-9b46234b7d4c-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177841    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177850    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98feb445-a93a-4db0-8130-9b46234b7d4c-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177858    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7nf\" (UniqueName: \"kubernetes.io/projected/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-kube-api-access-rp7nf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177867    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pstw\" (UniqueName: \"kubernetes.io/projected/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35-kube-api-access-8pstw\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.178055 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.177875    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 20:51:30.520206 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.520033    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:51:30.523109 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.523081    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-d65sv"]
Nov 01 20:51:30.631368 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.631299    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:51:30.635095 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.635068    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-z5gnd"]
Nov 01 20:51:30.730053 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.730022    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" path="/var/lib/kubelet/pods/6a7c06d8-e6eb-4575-a7d7-72457c2cabfb/volumes"
Nov 01 20:51:30.730461 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.730447    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" path="/var/lib/kubelet/pods/98feb445-a93a-4db0-8130-9b46234b7d4c/volumes"
Nov 01 20:51:30.742068 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.742046    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:51:30.749099 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:30.749073    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rszdb"]
Nov 01 20:51:32.729551 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:51:32.729487    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" path="/var/lib/kubelet/pods/7bd379cd-cb94-4dd0-b3a3-2fd36c53df35/volumes"
Nov 01 20:51:51.628368 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:51:51.628329    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:52:21.628721 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:52:21.628685    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:52:27.254992 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:52:27.254883    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:52:27.300147 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:52:27.300120    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:52:51.628285 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:52:51.628244    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:53:21.627646 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:53:21.627607    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:53:36.687564 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.687521    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 20:53:36.688002 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.687986    5066 cpu_manager.go:414] "containerMap: removing container" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688002    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688013    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688019    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688036    5066 cpu_manager.go:414] "containerMap: removing container" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688041    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy"
Nov 01 20:53:36.688049 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688050    5066 cpu_manager.go:414] "containerMap: removing container" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing"
Nov 01 20:53:36.688229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688055    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing"
Nov 01 20:53:36.688229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688131    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="789da027-7d9d-4ced-8165-ca18764e53f1" containerName="payload-processing"
Nov 01 20:53:36.688229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688145    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="98feb445-a93a-4db0-8130-9b46234b7d4c" containerName="istio-proxy"
Nov 01 20:53:36.688229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688152    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd379cd-cb94-4dd0-b3a3-2fd36c53df35" containerName="istio-proxy"
Nov 01 20:53:36.688229 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.688160    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7c06d8-e6eb-4575-a7d7-72457c2cabfb" containerName="istio-proxy"
Nov 01 20:53:36.691450 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.691432    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.745338 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.745308    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 20:53:36.789051 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.789004    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.789270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.789176    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.789270 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.789240    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.789394 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.789271    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjhl\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.795102 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795041    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 20:53:36.795292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795048    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 20:53:36.795292 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795057    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-433dc433b175\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795072    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795088    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-0d210cf9f3e2\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795130    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-1c1bf2169d38\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795132    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795334    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\""
Nov 01 20:53:36.795511 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.795142    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-2ab5a3fce61c\""
Nov 01 20:53:36.799213 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.799191    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 20:53:36.890130 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.890095    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.890130 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.890136    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjhl\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.890392 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.890172    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.890392 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.890239    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.891033 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.891003    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.891135 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.891017    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.893158 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.893132    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:36.946223 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:36.946139    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjhl\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:37.005218 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.005178    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:37.378690 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.378659    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 20:53:37.380796 ip-10-0-142-223 kubenswrapper[5066]: W1101 20:53:37.380769    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6828ead0_11fe_4870_86af_9cfd0ae14782.slice/crio-1c15eaea9ba9da2369f53e2dfa5c0bfe9f061cdcdbcc9332386a5f3494965160 WatchSource:0}: Error finding container 1c15eaea9ba9da2369f53e2dfa5c0bfe9f061cdcdbcc9332386a5f3494965160: Status 404 returned error can't find the container with id 1c15eaea9ba9da2369f53e2dfa5c0bfe9f061cdcdbcc9332386a5f3494965160
Nov 01 20:53:37.732072 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.732036    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" event={"ID":"6828ead0-11fe-4870-86af-9cfd0ae14782","Type":"ContainerStarted","Data":"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"}
Nov 01 20:53:37.732518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.732086    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" event={"ID":"6828ead0-11fe-4870-86af-9cfd0ae14782","Type":"ContainerStarted","Data":"1c15eaea9ba9da2369f53e2dfa5c0bfe9f061cdcdbcc9332386a5f3494965160"}
Nov 01 20:53:37.732518 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.732160    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:37.847277 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:37.847209    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" podStartSLOduration=1.847187458 podStartE2EDuration="1.847187458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:37.843912062 +0000 UTC m=+3071.228809653" watchObservedRunningTime="2026-11-01 20:53:37.847187458 +0000 UTC m=+3071.232085047"
Nov 01 20:53:48.740818 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:53:48.740784    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 20:53:51.628159 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:53:51.628119    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:54:21.627654 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:54:21.627574    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:54:51.627811 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:54:51.627763    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:55:21.627663 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:55:21.627625    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:55:51.628417 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:55:51.628332    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:56:21.627408 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:56:21.627372    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:56:51.626566 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:56:51.626529    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:57:21.628908 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:57:21.628824    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:57:27.291031 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:57:27.290928    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:57:27.345025 ip-10-0-142-223 kubenswrapper[5066]: I1101 20:57:27.344999    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 20:57:51.629457 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:57:51.629417    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:58:21.628143 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:58:21.628097    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:58:51.631910 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:58:51.631832    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:59:21.628173 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:59:21.628135    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 20:59:51.628114 ip-10-0-142-223 kubenswrapper[5066]: E1101 20:59:51.628074    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:00:21.514365 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.514321    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:21.518232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.518206    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.608255 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.608213    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9n7\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.608434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.608272    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.608434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.608404    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.608537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.608507    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.623449 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.623411    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:21.630628 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:00:21.630592    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:00:21.709863 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.709829    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.710048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.710010    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9n7\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.710095 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.710057    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.710131 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.710106    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.710609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.710592    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.710897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.710876    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.713072 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.713045    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.766418 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.766350    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9n7\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:21.833639 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:21.833606    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:22.205701 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:22.205669    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:22.207954 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:00:22.207923    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca48fbce_2a0d_4926_a2c2_af781ddb4522.slice/crio-13f8caccdc110f327eefaead0c1a9e0ee1f274df92ba298caa065d72ee33d784 WatchSource:0}: Error finding container 13f8caccdc110f327eefaead0c1a9e0ee1f274df92ba298caa065d72ee33d784: Status 404 returned error can't find the container with id 13f8caccdc110f327eefaead0c1a9e0ee1f274df92ba298caa065d72ee33d784
Nov 01 21:00:22.928779 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:22.928747    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" event={"ID":"ca48fbce-2a0d-4926-a2c2-af781ddb4522","Type":"ContainerStarted","Data":"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"}
Nov 01 21:00:22.929225 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:22.928796    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" event={"ID":"ca48fbce-2a0d-4926-a2c2-af781ddb4522","Type":"ContainerStarted","Data":"13f8caccdc110f327eefaead0c1a9e0ee1f274df92ba298caa065d72ee33d784"}
Nov 01 21:00:22.929225 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:22.928881    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:23.057665 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:23.057605    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" podStartSLOduration=2.057590119 podStartE2EDuration="2.057590119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:23.055917604 +0000 UTC m=+3476.440815195" watchObservedRunningTime="2026-11-01 21:00:23.057590119 +0000 UTC m=+3476.442487709"
Nov 01 21:00:33.937524 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:33.937451    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:43.935290 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:43.935244    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:46.068935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:46.068898    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:46.070053 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:46.070025    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:00:48.595135 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.595100    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:00:48.599005 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.598983    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.704573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.704529    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:00:48.778654 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.778616    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvw89\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.778868 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.778667    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.778868 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.778756    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.778868 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.778858    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.880483 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.880385    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.880699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.880548    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvw89\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.880699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.880598    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.880699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.880630    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.881223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.881191    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.881223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.881207    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.883316 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.883294    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:48.937055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:48.937023    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvw89\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:49.212762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:49.212672    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:49.587736 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:49.587708    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:00:49.589756 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:00:49.589718    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cb8815_0ef8_4a70_bdfc_ab05f264a210.slice/crio-602dd970b8ef85fed7461aa3ff0d8af443e4928d019d849d13ae6ff46ae56120 WatchSource:0}: Error finding container 602dd970b8ef85fed7461aa3ff0d8af443e4928d019d849d13ae6ff46ae56120: Status 404 returned error can't find the container with id 602dd970b8ef85fed7461aa3ff0d8af443e4928d019d849d13ae6ff46ae56120
Nov 01 21:00:50.086053 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:50.086011    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" event={"ID":"92cb8815-0ef8-4a70-bdfc-ab05f264a210","Type":"ContainerStarted","Data":"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"}
Nov 01 21:00:50.086553 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:50.086077    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" event={"ID":"92cb8815-0ef8-4a70-bdfc-ab05f264a210","Type":"ContainerStarted","Data":"602dd970b8ef85fed7461aa3ff0d8af443e4928d019d849d13ae6ff46ae56120"}
Nov 01 21:00:50.086553 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:50.086257    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:00:50.201329 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:50.201261    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" podStartSLOduration=2.201239738 podStartE2EDuration="2.201239738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:50.197923497 +0000 UTC m=+3503.582821090" watchObservedRunningTime="2026-11-01 21:00:50.201239738 +0000 UTC m=+3503.586137329"
Nov 01 21:00:51.070078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.070026    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing" containerID="cri-o://60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07" gracePeriod=25
Nov 01 21:00:51.332346 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.332283    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:51.509760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.509726    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\") pod \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") "
Nov 01 21:00:51.509965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.509796    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\") pod \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") "
Nov 01 21:00:51.509965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.509840    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\") pod \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") "
Nov 01 21:00:51.509965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.509925    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\") pod \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\" (UID: \"ca48fbce-2a0d-4926-a2c2-af781ddb4522\") "
Nov 01 21:00:51.510270 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.510237    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume" pod "ca48fbce-2a0d-4926-a2c2-af781ddb4522" (UID: "ca48fbce-2a0d-4926-a2c2-af781ddb4522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:51.510382 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.510279    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume" pod "ca48fbce-2a0d-4926-a2c2-af781ddb4522" (UID: "ca48fbce-2a0d-4926-a2c2-af781ddb4522"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:51.512138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.512096    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials" pod "ca48fbce-2a0d-4926-a2c2-af781ddb4522" (UID: "ca48fbce-2a0d-4926-a2c2-af781ddb4522"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:51.512228 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.512172    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7" pod "ca48fbce-2a0d-4926-a2c2-af781ddb4522" (UID: "ca48fbce-2a0d-4926-a2c2-af781ddb4522"). InnerVolumeSpecName "kube-api-access-bs9n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:51.611684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.611609    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:51.611684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.611637    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9n7\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-kube-api-access-bs9n7\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:51.611684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.611648    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca48fbce-2a0d-4926-a2c2-af781ddb4522-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:51.611684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:51.611657    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca48fbce-2a0d-4926-a2c2-af781ddb4522-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:00:51.628078 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:00:51.628040    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:00:52.098835 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.098802    5066 generic.go:356] "Generic (PLEG): container finished" podID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerID="60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07" exitCode=0
Nov 01 21:00:52.099048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.098849    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"
Nov 01 21:00:52.099048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.098881    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" event={"ID":"ca48fbce-2a0d-4926-a2c2-af781ddb4522","Type":"ContainerDied","Data":"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"}
Nov 01 21:00:52.099048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.098941    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw" event={"ID":"ca48fbce-2a0d-4926-a2c2-af781ddb4522","Type":"ContainerDied","Data":"13f8caccdc110f327eefaead0c1a9e0ee1f274df92ba298caa065d72ee33d784"}
Nov 01 21:00:52.099048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.098969    5066 scope.go:122] "RemoveContainer" containerID="60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"
Nov 01 21:00:52.108878 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.108848    5066 scope.go:122] "RemoveContainer" containerID="60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"
Nov 01 21:00:52.109456 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:00:52.109167    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07\": container with ID starting with 60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07 not found: ID does not exist" containerID="60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"
Nov 01 21:00:52.109456 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.109202    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07"} err="failed to get container status \"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07\": rpc error: code = NotFound desc = could not find container \"60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07\": container with ID starting with 60d25f382ba4815abe876ee9fd991d69145e29d16d099803cb986bd93d238e07 not found: ID does not exist"
Nov 01 21:00:52.261792 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.261754    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:52.264970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.264944    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-56cb47gbw"]
Nov 01 21:00:52.730121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:00:52.730085    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" path="/var/lib/kubelet/pods/ca48fbce-2a0d-4926-a2c2-af781ddb4522/volumes"
Nov 01 21:01:01.094851 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:01.094814    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:01:02.171822 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:02.171788    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:01:02.173062 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:02.173034    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:01:07.172232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.172169    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing" containerID="cri-o://63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b" gracePeriod=25
Nov 01 21:01:07.430185 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.430122    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:01:07.451448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451423    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\") pod \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") "
Nov 01 21:01:07.451606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451459    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\") pod \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") "
Nov 01 21:01:07.451606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451539    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\") pod \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") "
Nov 01 21:01:07.451606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451576    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\") pod \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\" (UID: \"92cb8815-0ef8-4a70-bdfc-ab05f264a210\") "
Nov 01 21:01:07.451865 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451836    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume" pod "92cb8815-0ef8-4a70-bdfc-ab05f264a210" (UID: "92cb8815-0ef8-4a70-bdfc-ab05f264a210"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:01:07.452012 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.451987    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume" pod "92cb8815-0ef8-4a70-bdfc-ab05f264a210" (UID: "92cb8815-0ef8-4a70-bdfc-ab05f264a210"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:01:07.453595 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.453569    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials" pod "92cb8815-0ef8-4a70-bdfc-ab05f264a210" (UID: "92cb8815-0ef8-4a70-bdfc-ab05f264a210"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:01:07.453710 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.453696    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89" pod "92cb8815-0ef8-4a70-bdfc-ab05f264a210" (UID: "92cb8815-0ef8-4a70-bdfc-ab05f264a210"). InnerVolumeSpecName "kube-api-access-hvw89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:01:07.552787 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.552754    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:07.552787 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.552783    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvw89\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-kube-api-access-hvw89\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:07.552958 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.552798    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92cb8815-0ef8-4a70-bdfc-ab05f264a210-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:07.552958 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:07.552810    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92cb8815-0ef8-4a70-bdfc-ab05f264a210-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:08.195531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.195486    5066 generic.go:356] "Generic (PLEG): container finished" podID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerID="63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b" exitCode=0
Nov 01 21:01:08.195938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.195544    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"
Nov 01 21:01:08.195938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.195573    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" event={"ID":"92cb8815-0ef8-4a70-bdfc-ab05f264a210","Type":"ContainerDied","Data":"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"}
Nov 01 21:01:08.195938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.195632    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n" event={"ID":"92cb8815-0ef8-4a70-bdfc-ab05f264a210","Type":"ContainerDied","Data":"602dd970b8ef85fed7461aa3ff0d8af443e4928d019d849d13ae6ff46ae56120"}
Nov 01 21:01:08.195938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.195661    5066 scope.go:122] "RemoveContainer" containerID="63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"
Nov 01 21:01:08.205448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.205430    5066 scope.go:122] "RemoveContainer" containerID="63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"
Nov 01 21:01:08.205704 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:01:08.205685    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b\": container with ID starting with 63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b not found: ID does not exist" containerID="63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"
Nov 01 21:01:08.205767 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.205711    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b"} err="failed to get container status \"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b\": rpc error: code = NotFound desc = could not find container \"63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b\": container with ID starting with 63759e50ea33e9f55a61fa0fa3e625c50b494954a9e93e69849461985a2dcf2b not found: ID does not exist"
Nov 01 21:01:08.362211 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.362176    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:01:08.408018 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.407985    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db876g8n"]
Nov 01 21:01:08.730096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:08.730064    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" path="/var/lib/kubelet/pods/92cb8815-0ef8-4a70-bdfc-ab05f264a210/volumes"
Nov 01 21:01:14.256644 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:14.256609    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 21:01:19.257355 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.257316    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" podUID="6828ead0-11fe-4870-86af-9cfd0ae14782" containerName="payload-processing" containerID="cri-o://ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4" gracePeriod=25
Nov 01 21:01:19.512749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.512687    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 21:01:19.575305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575276    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\") pod \"6828ead0-11fe-4870-86af-9cfd0ae14782\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") "
Nov 01 21:01:19.575469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575326    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\") pod \"6828ead0-11fe-4870-86af-9cfd0ae14782\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") "
Nov 01 21:01:19.575469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575386    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\") pod \"6828ead0-11fe-4870-86af-9cfd0ae14782\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") "
Nov 01 21:01:19.575469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575412    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\") pod \"6828ead0-11fe-4870-86af-9cfd0ae14782\" (UID: \"6828ead0-11fe-4870-86af-9cfd0ae14782\") "
Nov 01 21:01:19.575770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575740    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume" pod "6828ead0-11fe-4870-86af-9cfd0ae14782" (UID: "6828ead0-11fe-4870-86af-9cfd0ae14782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:01:19.575843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575807    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume" pod "6828ead0-11fe-4870-86af-9cfd0ae14782" (UID: "6828ead0-11fe-4870-86af-9cfd0ae14782"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:01:19.576000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.575978    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:19.576082 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.576005    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6828ead0-11fe-4870-86af-9cfd0ae14782-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:19.577543 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.577521    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials" pod "6828ead0-11fe-4870-86af-9cfd0ae14782" (UID: "6828ead0-11fe-4870-86af-9cfd0ae14782"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:01:19.577655 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.577554    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl" pod "6828ead0-11fe-4870-86af-9cfd0ae14782" (UID: "6828ead0-11fe-4870-86af-9cfd0ae14782"). InnerVolumeSpecName "kube-api-access-6xjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:01:19.676729 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.676700    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:19.676729 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:19.676726    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjhl\" (UniqueName: \"kubernetes.io/projected/6828ead0-11fe-4870-86af-9cfd0ae14782-kube-api-access-6xjhl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:01:20.264221 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.264178    5066 generic.go:356] "Generic (PLEG): container finished" podID="6828ead0-11fe-4870-86af-9cfd0ae14782" containerID="ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4" exitCode=0
Nov 01 21:01:20.264657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.264237    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"
Nov 01 21:01:20.264657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.264239    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" event={"ID":"6828ead0-11fe-4870-86af-9cfd0ae14782","Type":"ContainerDied","Data":"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"}
Nov 01 21:01:20.264657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.264369    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr" event={"ID":"6828ead0-11fe-4870-86af-9cfd0ae14782","Type":"ContainerDied","Data":"1c15eaea9ba9da2369f53e2dfa5c0bfe9f061cdcdbcc9332386a5f3494965160"}
Nov 01 21:01:20.264657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.264393    5066 scope.go:122] "RemoveContainer" containerID="ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"
Nov 01 21:01:20.274665 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.274370    5066 scope.go:122] "RemoveContainer" containerID="ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"
Nov 01 21:01:20.274822 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:01:20.274788    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4\": container with ID starting with ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4 not found: ID does not exist" containerID="ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"
Nov 01 21:01:20.274898 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.274817    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4"} err="failed to get container status \"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4\": rpc error: code = NotFound desc = could not find container \"ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4\": container with ID starting with ae30d0c131974eed44fefcafb08bd13f9ee9a270982bc6b76db4259010e130c4 not found: ID does not exist"
Nov 01 21:01:20.427201 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.427164    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 21:01:20.476762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.476729    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-547fpcflr"]
Nov 01 21:01:20.730502 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:20.730468    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6828ead0-11fe-4870-86af-9cfd0ae14782" path="/var/lib/kubelet/pods/6828ead0-11fe-4870-86af-9cfd0ae14782/volumes"
Nov 01 21:01:21.628447 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:01:21.628411    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:01:33.834521 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.834476    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:01:33.834988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.834941    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing"
Nov 01 21:01:33.834988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.834952    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing"
Nov 01 21:01:33.834988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.834970    5066 cpu_manager.go:414] "containerMap: removing container" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing"
Nov 01 21:01:33.834988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.834975    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing"
Nov 01 21:01:33.835121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.835006    5066 cpu_manager.go:414] "containerMap: removing container" podUID="6828ead0-11fe-4870-86af-9cfd0ae14782" containerName="payload-processing"
Nov 01 21:01:33.835121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.835012    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6828ead0-11fe-4870-86af-9cfd0ae14782" containerName="payload-processing"
Nov 01 21:01:33.835121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.835078    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="6828ead0-11fe-4870-86af-9cfd0ae14782" containerName="payload-processing"
Nov 01 21:01:33.835121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.835087    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca48fbce-2a0d-4926-a2c2-af781ddb4522" containerName="payload-processing"
Nov 01 21:01:33.835121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.835094    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cb8815-0ef8-4a70-bdfc-ab05f264a210" containerName="payload-processing"
Nov 01 21:01:33.838302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.838285    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:33.942264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942233    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 21:01:33.942447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942281    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 21:01:33.942447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942276    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9795e2cbcfc6\""
Nov 01 21:01:33.942447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942298    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 21:01:33.942447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942299    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 21:01:33.942447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.942283    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-9wq7t\""
Nov 01 21:01:33.944682 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:33.944662    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:01:34.021150 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.021117    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.021323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.021173    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.021323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.021236    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.021323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.021267    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgn8g\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.122548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.122423    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.122548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.122467    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.122548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.122490    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgn8g\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.122833 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.122691    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.123242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.123217    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.123442 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.123417    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.125121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.125097    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.178958 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.178926    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgn8g\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\") pod \"payload-processing-external-model-9944c884c-qplv2\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") " pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.451476 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.451392    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:34.830203 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:34.830171    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:01:34.832421 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:01:34.832390    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34580354_8470_4fa8_9f00_2d8adf3bb392.slice/crio-e53597f01c91379c7eb0978d01da5bf464c2af19175287aed1c353e487c3dee4 WatchSource:0}: Error finding container e53597f01c91379c7eb0978d01da5bf464c2af19175287aed1c353e487c3dee4: Status 404 returned error can't find the container with id e53597f01c91379c7eb0978d01da5bf464c2af19175287aed1c353e487c3dee4
Nov 01 21:01:35.351364 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:35.351327    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" event={"ID":"34580354-8470-4fa8-9f00-2d8adf3bb392","Type":"ContainerStarted","Data":"8cfdb7442c193b60a24c5af21ba4f15d6ba10199f06bd1c030cbbb3551a1a05c"}
Nov 01 21:01:35.351731 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:35.351419    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" event={"ID":"34580354-8470-4fa8-9f00-2d8adf3bb392","Type":"ContainerStarted","Data":"e53597f01c91379c7eb0978d01da5bf464c2af19175287aed1c353e487c3dee4"}
Nov 01 21:01:35.351731 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:35.351545    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:35.465860 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:35.465808    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" podStartSLOduration=2.465792657 podStartE2EDuration="2.465792657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:35.464035442 +0000 UTC m=+3548.848933032" watchObservedRunningTime="2026-11-01 21:01:35.465792657 +0000 UTC m=+3548.850690247"
Nov 01 21:01:46.360187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:46.360158    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:01:51.629348 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:01:51.629310    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:01:59.351002 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.350964    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:01:59.354858 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.354840    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.406922 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.406886    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 21:01:59.459297 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.459260    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:01:59.469743 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.469717    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.469890 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.469755    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.469890 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.469874    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.469906    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7kd\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.469989    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.470051    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.470085    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470338 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.470172    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.470338 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.470202    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571003 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.570974    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571003 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571008    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571206 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571138    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571206 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571175    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7kd\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571222    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571248    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571507 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571469    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571710 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571692    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571541    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571729    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571599    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.571910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.571777    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.572063 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.572043    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.572125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.572108    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.573487 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.573461    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.573715 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.573698    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.626844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.626787    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.627037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.627020    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7kd\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dx22m\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:01:59.675461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:01:59.675427    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:02:00.102769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.102741    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:02:00.104683 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:02:00.104656    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1747a8_0a08_47a3_9e19_04a6d38b9621.slice/crio-f48c789cf0d2d5978a0d4d196553214f19a4a8a8e9c2bab439bdd156f24b2b44 WatchSource:0}: Error finding container f48c789cf0d2d5978a0d4d196553214f19a4a8a8e9c2bab439bdd156f24b2b44: Status 404 returned error can't find the container with id f48c789cf0d2d5978a0d4d196553214f19a4a8a8e9c2bab439bdd156f24b2b44
Nov 01 21:02:00.106905 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.106874    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:00.106986 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.106952    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:00.107028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.106988    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:00.488268 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.488227    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" event={"ID":"7b1747a8-0a08-47a3-9e19-04a6d38b9621","Type":"ContainerStarted","Data":"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"}
Nov 01 21:02:00.488700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.488327    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" event={"ID":"7b1747a8-0a08-47a3-9e19-04a6d38b9621","Type":"ContainerStarted","Data":"f48c789cf0d2d5978a0d4d196553214f19a4a8a8e9c2bab439bdd156f24b2b44"}
Nov 01 21:02:00.607340 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.607271    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podStartSLOduration=1.607252086 podStartE2EDuration="1.607252086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:00.603265598 +0000 UTC m=+3573.988163188" watchObservedRunningTime="2026-11-01 21:02:00.607252086 +0000 UTC m=+3573.992149676"
Nov 01 21:02:00.677293 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:00.677258    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:02:01.677823 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:01.677787    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:01.678391 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:01.678359    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:02.684519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:02.676549    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:02.684519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:02.676606    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:03.680996 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:03.680957    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:03.681280 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:03.681252    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:04.678964 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:04.678928    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:04.679613 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:04.679579    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:05.678862 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:05.678822    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:05.679284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:05.679252    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:06.676227 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:06.676191    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:06.676422 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:06.676246    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:07.675824 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:07.675790    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dx22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 21:02:07.676205 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:07.675844    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 21:02:09.066946 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:09.066912    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:02:09.067705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:09.067673    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:02:09.068669 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:09.068647    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:02:17.498163 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:17.498130    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:02:21.630098 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:21.630062    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:02:22.499378 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.499318    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" podUID="34580354-8470-4fa8-9f00-2d8adf3bb392" containerName="payload-processing" containerID="cri-o://8cfdb7442c193b60a24c5af21ba4f15d6ba10199f06bd1c030cbbb3551a1a05c" gracePeriod=25
Nov 01 21:02:22.633067 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.633037    5066 generic.go:356] "Generic (PLEG): container finished" podID="34580354-8470-4fa8-9f00-2d8adf3bb392" containerID="8cfdb7442c193b60a24c5af21ba4f15d6ba10199f06bd1c030cbbb3551a1a05c" exitCode=0
Nov 01 21:02:22.633468 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.633096    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" event={"ID":"34580354-8470-4fa8-9f00-2d8adf3bb392","Type":"ContainerDied","Data":"8cfdb7442c193b60a24c5af21ba4f15d6ba10199f06bd1c030cbbb3551a1a05c"}
Nov 01 21:02:22.792902 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.792872    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:02:22.808112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808064    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\") pod \"34580354-8470-4fa8-9f00-2d8adf3bb392\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") "
Nov 01 21:02:22.808112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808129    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\") pod \"34580354-8470-4fa8-9f00-2d8adf3bb392\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") "
Nov 01 21:02:22.808338 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808176    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\") pod \"34580354-8470-4fa8-9f00-2d8adf3bb392\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") "
Nov 01 21:02:22.808338 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808195    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\") pod \"34580354-8470-4fa8-9f00-2d8adf3bb392\" (UID: \"34580354-8470-4fa8-9f00-2d8adf3bb392\") "
Nov 01 21:02:22.808622 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808595    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume" pod "34580354-8470-4fa8-9f00-2d8adf3bb392" (UID: "34580354-8470-4fa8-9f00-2d8adf3bb392"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:22.808781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.808612    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume" pod "34580354-8470-4fa8-9f00-2d8adf3bb392" (UID: "34580354-8470-4fa8-9f00-2d8adf3bb392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:22.810650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.810623    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g" pod "34580354-8470-4fa8-9f00-2d8adf3bb392" (UID: "34580354-8470-4fa8-9f00-2d8adf3bb392"). InnerVolumeSpecName "kube-api-access-mgn8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:22.811058 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.811028    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials" pod "34580354-8470-4fa8-9f00-2d8adf3bb392" (UID: "34580354-8470-4fa8-9f00-2d8adf3bb392"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:22.911950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.911915    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:22.911950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.911950    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgn8g\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-kube-api-access-mgn8g\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:22.912214 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.911964    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34580354-8470-4fa8-9f00-2d8adf3bb392-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:22.912214 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:22.911977    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34580354-8470-4fa8-9f00-2d8adf3bb392-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:23.639950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:23.639918    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2" event={"ID":"34580354-8470-4fa8-9f00-2d8adf3bb392","Type":"ContainerDied","Data":"e53597f01c91379c7eb0978d01da5bf464c2af19175287aed1c353e487c3dee4"}
Nov 01 21:02:23.639950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:23.639947    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9944c884c-qplv2"
Nov 01 21:02:23.640431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:23.639979    5066 scope.go:122] "RemoveContainer" containerID="8cfdb7442c193b60a24c5af21ba4f15d6ba10199f06bd1c030cbbb3551a1a05c"
Nov 01 21:02:23.803567 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:23.803531    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:02:23.852863 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:23.852826    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9944c884c-qplv2"]
Nov 01 21:02:24.731031 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:24.730988    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34580354-8470-4fa8-9f00-2d8adf3bb392" path="/var/lib/kubelet/pods/34580354-8470-4fa8-9f00-2d8adf3bb392/volumes"
Nov 01 21:02:27.325232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:27.325130    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:02:27.386334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:27.386308    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:02:44.687897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.687860    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:02:44.688582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.688430    5066 cpu_manager.go:414] "containerMap: removing container" podUID="34580354-8470-4fa8-9f00-2d8adf3bb392" containerName="payload-processing"
Nov 01 21:02:44.688582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.688447    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34580354-8470-4fa8-9f00-2d8adf3bb392" containerName="payload-processing"
Nov 01 21:02:44.688582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.688581    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="34580354-8470-4fa8-9f00-2d8adf3bb392" containerName="payload-processing"
Nov 01 21:02:44.691904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.691880    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.801977 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.801934    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:02:44.852508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852457    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852685 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852525    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852685 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852587    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn25b\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852685 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852639    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852791 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852723    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852791 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852762    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852859 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852793    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852911 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.852945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.852926    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954377 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954288    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954569 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954429    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954569 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954479    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954569 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954542    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954587    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954618    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954681    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954706    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.954928 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.954749    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn25b\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.955038 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.955011    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.955118 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.955090    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.955368 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.955159    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.955368 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.955194    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.955654 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.955629    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.957537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.957515    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:44.957682 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:44.957665    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:45.011937 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.011901    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn25b\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:45.012356 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.012333    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cr2pz\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:45.304941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.304843    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:45.744932 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.744901    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:02:45.746530 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:02:45.746475    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4047a62_1230_47bb_9b80_7ed53390deb2.slice/crio-06849851ab37a22b91e7aa29a24795db2349fd0d1eb09106239c380fd47ac116 WatchSource:0}: Error finding container 06849851ab37a22b91e7aa29a24795db2349fd0d1eb09106239c380fd47ac116: Status 404 returned error can't find the container with id 06849851ab37a22b91e7aa29a24795db2349fd0d1eb09106239c380fd47ac116
Nov 01 21:02:45.748886 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.748853    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:45.748989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.748932    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:45.749052 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.748988    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:02:45.770096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.770067    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" event={"ID":"b4047a62-1230-47bb-9b80-7ed53390deb2","Type":"ContainerStarted","Data":"06849851ab37a22b91e7aa29a24795db2349fd0d1eb09106239c380fd47ac116"}
Nov 01 21:02:45.817123 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.817087    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:45.820887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.820861    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.874506 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.874425    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.874770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.874539    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfgf\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.874770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.874603    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.874770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.874662    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.875127 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.875098    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 21:02:45.875248 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.875226    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 21:02:45.875367 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.875346    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 21:02:45.875437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.875406    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 21:02:45.880241 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.880213    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:45.908733 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.908701    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4kfcm\""
Nov 01 21:02:45.908733 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.908718    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-25a736a860a0\""
Nov 01 21:02:45.975987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.975943    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.976178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.976028    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfgf\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.976178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.976066    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.976178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.976127    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.976937 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.976909    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.977118 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.976997    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:45.978822 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:45.978796    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:46.040853 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.040780    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfgf\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\") pod \"payload-processing-external-model-bc6848fd9-vzk6x\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") " pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:46.157151 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.157105    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:46.554442 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.551670    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:46.554599 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:02:46.554474    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f50c09_cda6_4e95_9500_a3859493ebb9.slice/crio-030911899c37a7ba3ac2f9ec2b27cdcd93e6a17af4093def61c0d03c3a75059f WatchSource:0}: Error finding container 030911899c37a7ba3ac2f9ec2b27cdcd93e6a17af4093def61c0d03c3a75059f: Status 404 returned error can't find the container with id 030911899c37a7ba3ac2f9ec2b27cdcd93e6a17af4093def61c0d03c3a75059f
Nov 01 21:02:46.801577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.801098    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" event={"ID":"b4047a62-1230-47bb-9b80-7ed53390deb2","Type":"ContainerStarted","Data":"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"}
Nov 01 21:02:46.807678 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.806722    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" event={"ID":"64f50c09-cda6-4e95-9500-a3859493ebb9","Type":"ContainerStarted","Data":"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"}
Nov 01 21:02:46.807678 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.806796    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" event={"ID":"64f50c09-cda6-4e95-9500-a3859493ebb9","Type":"ContainerStarted","Data":"030911899c37a7ba3ac2f9ec2b27cdcd93e6a17af4093def61c0d03c3a75059f"}
Nov 01 21:02:46.807678 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.807463    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:46.920730 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:46.918723    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podStartSLOduration=2.918706901 podStartE2EDuration="2.918706901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:46.917243937 +0000 UTC m=+3620.302141529" watchObservedRunningTime="2026-11-01 21:02:46.918706901 +0000 UTC m=+3620.303604492"
Nov 01 21:02:47.040295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:47.040232    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" podStartSLOduration=2.040213625 podStartE2EDuration="2.040213625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:47.038570732 +0000 UTC m=+3620.423468324" watchObservedRunningTime="2026-11-01 21:02:47.040213625 +0000 UTC m=+3620.425111259"
Nov 01 21:02:47.305115 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:47.305064    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:47.306944 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:47.306910    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:47.307104 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:47.306970    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:48.014106 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.014065    5066 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-25a736a860a0: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.014106 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.014091    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.014689 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.014134    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials podName:64f50c09-cda6-4e95-9500-a3859493ebb9 nodeName:}" failed. No retries permitted until 2026-11-01 21:02:48.514118532 +0000 UTC m=+3621.899016100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials") pod "payload-processing-external-model-bc6848fd9-vzk6x" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9") : secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.305806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:48.305710    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:48.305806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:48.305776    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:48.517569 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.517531    5066 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-25a736a860a0: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.517569 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.517560    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.517800 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:48.517612    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials podName:64f50c09-cda6-4e95-9500-a3859493ebb9 nodeName:}" failed. No retries permitted until 2026-11-01 21:02:49.517597776 +0000 UTC m=+3622.902495357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials") pod "payload-processing-external-model-bc6848fd9-vzk6x" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9") : secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:48.806468 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:48.806410    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:49.305469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:49.305433    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:49.305907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:49.305526    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:49.526316 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:49.526277    5066 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-25a736a860a0: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:49.526316 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:49.526305    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:49.526617 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:49.526368    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials podName:64f50c09-cda6-4e95-9500-a3859493ebb9 nodeName:}" failed. No retries permitted until 2026-11-01 21:02:51.526350992 +0000 UTC m=+3624.911248573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials") pod "payload-processing-external-model-bc6848fd9-vzk6x" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9") : secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:50.305985 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:50.305947    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:50.306399 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:50.306003    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:51.305478 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:51.305441    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:51.305706 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:51.305541    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:51.546918 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:51.546885    5066 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-25a736a860a0: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:51.546918 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:51.546913    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x: secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:51.547334 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:51.546965    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials podName:64f50c09-cda6-4e95-9500-a3859493ebb9 nodeName:}" failed. No retries permitted until 2026-11-01 21:02:55.546949816 +0000 UTC m=+3628.931847400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials") pod "payload-processing-external-model-bc6848fd9-vzk6x" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9") : secret "agc-cred-25a736a860a0" not found
Nov 01 21:02:51.639195 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:51.639093    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:02:52.305834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:52.305793    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:52.306008 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:52.305860    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:53.305429 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:53.305390    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cr2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 01 21:02:53.305935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:53.305444    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 01 21:02:54.828467 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:54.828416    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" podUID="64f50c09-cda6-4e95-9500-a3859493ebb9" containerName="payload-processing" containerID="cri-o://c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2" gracePeriod=25
Nov 01 21:02:55.073288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.073239    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:55.074365 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.074345    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:55.075170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.075143    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:02:55.107423 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.107394    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:55.187348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187310    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\") pod \"64f50c09-cda6-4e95-9500-a3859493ebb9\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") "
Nov 01 21:02:55.187554 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187380    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\") pod \"64f50c09-cda6-4e95-9500-a3859493ebb9\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") "
Nov 01 21:02:55.187554 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187485    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\") pod \"64f50c09-cda6-4e95-9500-a3859493ebb9\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") "
Nov 01 21:02:55.187696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187648    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\") pod \"64f50c09-cda6-4e95-9500-a3859493ebb9\" (UID: \"64f50c09-cda6-4e95-9500-a3859493ebb9\") "
Nov 01 21:02:55.188259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187864    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume" pod "64f50c09-cda6-4e95-9500-a3859493ebb9" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:55.188259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.187882    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume" pod "64f50c09-cda6-4e95-9500-a3859493ebb9" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:02:55.188259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.188058    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:55.188259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.188078    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f50c09-cda6-4e95-9500-a3859493ebb9-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:55.189861 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.189832    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials" pod "64f50c09-cda6-4e95-9500-a3859493ebb9" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:55.189987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.189974    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf" pod "64f50c09-cda6-4e95-9500-a3859493ebb9" (UID: "64f50c09-cda6-4e95-9500-a3859493ebb9"). InnerVolumeSpecName "kube-api-access-kqfgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:02:55.289714 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.289680    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqfgf\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-kube-api-access-kqfgf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:55.289714 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.289711    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f50c09-cda6-4e95-9500-a3859493ebb9-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:02:55.522590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.522546    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:02:55.523322 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.523296    5066 cpu_manager.go:414] "containerMap: removing container" podUID="64f50c09-cda6-4e95-9500-a3859493ebb9" containerName="payload-processing"
Nov 01 21:02:55.523322 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.523322    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f50c09-cda6-4e95-9500-a3859493ebb9" containerName="payload-processing"
Nov 01 21:02:55.523546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.523470    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f50c09-cda6-4e95-9500-a3859493ebb9" containerName="payload-processing"
Nov 01 21:02:55.527411 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.527391    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.589023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.588987    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-rx28r\""
Nov 01 21:02:55.589023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.588987    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 21:02:55.589023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.588987    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-daf6244ee5be\""
Nov 01 21:02:55.589279 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.588987    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 21:02:55.589279 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.588995    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 21:02:55.589279 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.589067    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 21:02:55.594858 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.594833    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlvj\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.594970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.594867    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.594970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.594893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.595069 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.595053    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.616587 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.613823    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:02:55.696436 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.696401    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.696645 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.696455    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlvj\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.696645 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.696578    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.696645 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.696636    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.697294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.697266    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.697425 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.697304    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.699040 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.699010    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.752603 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.752568    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlvj\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.841708 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.841619    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:55.867014 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.866977    5066 generic.go:356] "Generic (PLEG): container finished" podID="64f50c09-cda6-4e95-9500-a3859493ebb9" containerID="c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2" exitCode=0
Nov 01 21:02:55.867170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.867032    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"
Nov 01 21:02:55.867170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.867066    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" event={"ID":"64f50c09-cda6-4e95-9500-a3859493ebb9","Type":"ContainerDied","Data":"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"}
Nov 01 21:02:55.867170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.867136    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x" event={"ID":"64f50c09-cda6-4e95-9500-a3859493ebb9","Type":"ContainerDied","Data":"030911899c37a7ba3ac2f9ec2b27cdcd93e6a17af4093def61c0d03c3a75059f"}
Nov 01 21:02:55.867351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.867173    5066 scope.go:122] "RemoveContainer" containerID="c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"
Nov 01 21:02:55.881864 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.881842    5066 scope.go:122] "RemoveContainer" containerID="c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"
Nov 01 21:02:55.882230 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:02:55.882205    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2\": container with ID starting with c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2 not found: ID does not exist" containerID="c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"
Nov 01 21:02:55.882311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:55.882248    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2"} err="failed to get container status \"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2\": rpc error: code = NotFound desc = could not find container \"c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2\": container with ID starting with c1a091bbebe1777cad5c42f80daa71561fbc5447db19c85a4af6890e61d88fd2 not found: ID does not exist"
Nov 01 21:02:56.031161 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.031125    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:56.035161 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.035132    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6848fd9-vzk6x"]
Nov 01 21:02:56.222622 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.222592    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:02:56.225611 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:02:56.225584    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode975cc52_b6e6_40bc_a90f_606bd736e7df.slice/crio-bd5da2aa43e76579d6293cc933003966b70f244a322a54d528ee1a94ed7b8972 WatchSource:0}: Error finding container bd5da2aa43e76579d6293cc933003966b70f244a322a54d528ee1a94ed7b8972: Status 404 returned error can't find the container with id bd5da2aa43e76579d6293cc933003966b70f244a322a54d528ee1a94ed7b8972
Nov 01 21:02:56.732510 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.732457    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f50c09-cda6-4e95-9500-a3859493ebb9" path="/var/lib/kubelet/pods/64f50c09-cda6-4e95-9500-a3859493ebb9/volumes"
Nov 01 21:02:56.877986 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.877947    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" event={"ID":"e975cc52-b6e6-40bc-a90f-606bd736e7df","Type":"ContainerStarted","Data":"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"}
Nov 01 21:02:56.878529 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.878010    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" event={"ID":"e975cc52-b6e6-40bc-a90f-606bd736e7df","Type":"ContainerStarted","Data":"bd5da2aa43e76579d6293cc933003966b70f244a322a54d528ee1a94ed7b8972"}
Nov 01 21:02:56.878914 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.878892    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:02:56.992200 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:56.991550    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" podStartSLOduration=1.991532012 podStartE2EDuration="1.991532012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:56.991526883 +0000 UTC m=+3630.376424474" watchObservedRunningTime="2026-11-01 21:02:56.991532012 +0000 UTC m=+3630.376429595"
Nov 01 21:02:59.798878 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.798844    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:02:59.802835 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.802812    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843111 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843078    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wv8\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843111 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843117    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843136    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843153    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843250    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843339    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843377    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843547 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843530    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.843648 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.843632    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.905261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.905220    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:02:59.944343 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944292    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944343 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944349    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944426    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944485    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944551    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wv8\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944581    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944605    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944755    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944795    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944808    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.944923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.944843    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.945187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.945163    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.945326 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.945305    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.945394 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.945332    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.946889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.946862    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:02:59.947205 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:02:59.947187    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:00.001109 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.001078    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:00.001348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.001323    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wv8\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5s667\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:00.115784 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.115686    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:00.541549 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.541519    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:03:00.543578 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:03:00.543547    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8f9686_1afa_4299_9583_d92a035a3c67.slice/crio-b3d3f53a3645ba6d69f5b39d822634b7ebe7a80d39393be7c12585cff1567bf9 WatchSource:0}: Error finding container b3d3f53a3645ba6d69f5b39d822634b7ebe7a80d39393be7c12585cff1567bf9: Status 404 returned error can't find the container with id b3d3f53a3645ba6d69f5b39d822634b7ebe7a80d39393be7c12585cff1567bf9
Nov 01 21:03:00.545801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.545770    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:00.545861 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.545854    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:00.545902 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.545881    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:03:00.913949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.913839    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" event={"ID":"be8f9686-1afa-4299-9583-d92a035a3c67","Type":"ContainerStarted","Data":"6afdc3ff99bd82a06f86f35fbafec5bb236056fd07cc4ea327b10329a58e1c34"}
Nov 01 21:03:00.913949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:00.913913    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" event={"ID":"be8f9686-1afa-4299-9583-d92a035a3c67","Type":"ContainerStarted","Data":"b3d3f53a3645ba6d69f5b39d822634b7ebe7a80d39393be7c12585cff1567bf9"}
Nov 01 21:03:01.035526 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:01.035440    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podStartSLOduration=2.035420942 podStartE2EDuration="2.035420942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:01.032761155 +0000 UTC m=+3634.417658767" watchObservedRunningTime="2026-11-01 21:03:01.035420942 +0000 UTC m=+3634.420318535"
Nov 01 21:03:01.116124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:01.116081    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:01.117589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:01.117561    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:01.117706 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:01.117617    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:02.116178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:02.116139    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:02.116620 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:02.116194    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:03.116968 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:03.116931    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:03.117363 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:03.116990    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:04.116026 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:04.115984    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:04.116256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:04.116046    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:05.116913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:05.116876    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:05.117413 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:05.116929    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:06.116458 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:06.116419    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:06.116644 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:06.116481    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:07.116303 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:07.116267    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:07.116807 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:07.116323    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:08.117077 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:08.116998    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 21:03:08.117077 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:08.117064    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 21:03:08.900879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:08.900848    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:03:09.554258 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:09.554224    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:09.554876 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:09.554847    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:09.555819 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:03:09.555804    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:03:21.628166 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:03:21.628129    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:03:51.627490 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:03:51.627451    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:04:09.967704 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:09.967670    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:04:14.968589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:14.968540    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" podUID="e975cc52-b6e6-40bc-a90f-606bd736e7df" containerName="payload-processing" containerID="cri-o://ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79" gracePeriod=25
Nov 01 21:04:15.218796 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.218742    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:04:15.345664 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.345629    5066 generic.go:356] "Generic (PLEG): container finished" podID="e975cc52-b6e6-40bc-a90f-606bd736e7df" containerID="ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79" exitCode=0
Nov 01 21:04:15.345831 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.345667    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"
Nov 01 21:04:15.345831 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.345708    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" event={"ID":"e975cc52-b6e6-40bc-a90f-606bd736e7df","Type":"ContainerDied","Data":"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"}
Nov 01 21:04:15.345831 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.345774    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8" event={"ID":"e975cc52-b6e6-40bc-a90f-606bd736e7df","Type":"ContainerDied","Data":"bd5da2aa43e76579d6293cc933003966b70f244a322a54d528ee1a94ed7b8972"}
Nov 01 21:04:15.345831 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.345807    5066 scope.go:122] "RemoveContainer" containerID="ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"
Nov 01 21:04:15.355168 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355147    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\") pod \"e975cc52-b6e6-40bc-a90f-606bd736e7df\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") "
Nov 01 21:04:15.355294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355233    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\") pod \"e975cc52-b6e6-40bc-a90f-606bd736e7df\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") "
Nov 01 21:04:15.355401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355379    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\") pod \"e975cc52-b6e6-40bc-a90f-606bd736e7df\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") "
Nov 01 21:04:15.355533 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355515    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\") pod \"e975cc52-b6e6-40bc-a90f-606bd736e7df\" (UID: \"e975cc52-b6e6-40bc-a90f-606bd736e7df\") "
Nov 01 21:04:15.355631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355577    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume" pod "e975cc52-b6e6-40bc-a90f-606bd736e7df" (UID: "e975cc52-b6e6-40bc-a90f-606bd736e7df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:04:15.355921 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355897    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume" pod "e975cc52-b6e6-40bc-a90f-606bd736e7df" (UID: "e975cc52-b6e6-40bc-a90f-606bd736e7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:04:15.356036 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.355965    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:15.359624 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.359604    5066 scope.go:122] "RemoveContainer" containerID="ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"
Nov 01 21:04:15.359728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.359625    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials" pod "e975cc52-b6e6-40bc-a90f-606bd736e7df" (UID: "e975cc52-b6e6-40bc-a90f-606bd736e7df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:15.359728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.359663    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj" pod "e975cc52-b6e6-40bc-a90f-606bd736e7df" (UID: "e975cc52-b6e6-40bc-a90f-606bd736e7df"). InnerVolumeSpecName "kube-api-access-fvlvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:04:15.359917 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:04:15.359896    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79\": container with ID starting with ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79 not found: ID does not exist" containerID="ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"
Nov 01 21:04:15.359974 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.359928    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79"} err="failed to get container status \"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79\": rpc error: code = NotFound desc = could not find container \"ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79\": container with ID starting with ba7bda2145422b161ec27f28fbceed533afabaa907a5776f1c3d3252b35ada79 not found: ID does not exist"
Nov 01 21:04:15.459437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.459397    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e975cc52-b6e6-40bc-a90f-606bd736e7df-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:15.459437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.459439    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlvj\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-kube-api-access-fvlvj\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:15.459705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.459470    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e975cc52-b6e6-40bc-a90f-606bd736e7df-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:04:15.765820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.765789    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:04:15.768923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:15.768898    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b86sxpf8"]
Nov 01 21:04:16.730574 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:16.730541    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e975cc52-b6e6-40bc-a90f-606bd736e7df" path="/var/lib/kubelet/pods/e975cc52-b6e6-40bc-a90f-606bd736e7df/volumes"
Nov 01 21:04:21.627878 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:04:21.627837    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:04:22.390315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.390255    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:04:22.391171 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.391145    5066 cpu_manager.go:414] "containerMap: removing container" podUID="e975cc52-b6e6-40bc-a90f-606bd736e7df" containerName="payload-processing"
Nov 01 21:04:22.391171 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.391171    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e975cc52-b6e6-40bc-a90f-606bd736e7df" containerName="payload-processing"
Nov 01 21:04:22.391380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.391281    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="e975cc52-b6e6-40bc-a90f-606bd736e7df" containerName="payload-processing"
Nov 01 21:04:22.395994 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.395967    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.449022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.448987    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:04:22.494963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.494932    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\""
Nov 01 21:04:22.494963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.494958    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dd468d87ff44\""
Nov 01 21:04:22.495243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.494932    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\""
Nov 01 21:04:22.495243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.494943    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\""
Nov 01 21:04:22.495243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.494932    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\""
Nov 01 21:04:22.497139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.497121    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-nnqcq\""
Nov 01 21:04:22.544190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.544159    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.544391 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.544285    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz852\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.544391 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.544318    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.544469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.544426    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.645970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.645888    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz852\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.645970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.645925    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.645970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.645962    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.646537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.646022    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.646875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.646849    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.646913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.646861    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.648978 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.648947    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.703187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.703152    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz852\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:22.708975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:22.708951    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:23.081698 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:23.081666    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:04:23.083984 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:04:23.083959    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d0e221_a880_4f55_ba0e_a745609a96b5.slice/crio-9f2bb74a9bf635e67a186ea7fb7d61e5f3b8ceacfc87a8c2dd5e2d86375e9a40 WatchSource:0}: Error finding container 9f2bb74a9bf635e67a186ea7fb7d61e5f3b8ceacfc87a8c2dd5e2d86375e9a40: Status 404 returned error can't find the container with id 9f2bb74a9bf635e67a186ea7fb7d61e5f3b8ceacfc87a8c2dd5e2d86375e9a40
Nov 01 21:04:23.402991 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:23.402901    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" event={"ID":"32d0e221-a880-4f55-ba0e-a745609a96b5","Type":"ContainerStarted","Data":"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"}
Nov 01 21:04:23.402991 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:23.402957    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" event={"ID":"32d0e221-a880-4f55-ba0e-a745609a96b5","Type":"ContainerStarted","Data":"9f2bb74a9bf635e67a186ea7fb7d61e5f3b8ceacfc87a8c2dd5e2d86375e9a40"}
Nov 01 21:04:23.402991 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:23.402982    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:23.516183 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:23.516120    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" podStartSLOduration=1.516103802 podStartE2EDuration="1.516103802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:23.515679176 +0000 UTC m=+3716.900576767" watchObservedRunningTime="2026-11-01 21:04:23.516103802 +0000 UTC m=+3716.901001393"
Nov 01 21:04:34.412340 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:04:34.412309    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:04:51.627155 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:04:51.627078    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:05:21.635341 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:05:21.635295    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:05:35.317514 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.317451    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:05:35.321373 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.321348    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.374342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.374012    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-14958f7d1994\""
Nov 01 21:05:35.377367 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.377326    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:05:35.423582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.423534    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.423781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.423688    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.423781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.423726    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.423907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.423815    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcg5c\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.525257 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.525209    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.525440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.525270    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.525440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.525297    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcg5c\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.525440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.525363    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.526024 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.525998    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.526268 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.526240    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.528008 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.527984    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.581816 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.581735    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcg5c\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:35.635474 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:35.635439    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:36.016771 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:36.016737    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:05:36.017993 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:05:36.017969    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be7914_c4ad_46e9_9eaf_707bac1e5199.slice/crio-52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e WatchSource:0}: Error finding container 52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e: Status 404 returned error can't find the container with id 52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e
Nov 01 21:05:36.833706 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:36.833669    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" event={"ID":"c7be7914-c4ad-46e9-9eaf-707bac1e5199","Type":"ContainerStarted","Data":"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"}
Nov 01 21:05:36.834116 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:36.833723    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" event={"ID":"c7be7914-c4ad-46e9-9eaf-707bac1e5199","Type":"ContainerStarted","Data":"52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e"}
Nov 01 21:05:36.834116 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:36.833754    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:36.949631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:36.949578    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" podStartSLOduration=1.949565646 podStartE2EDuration="1.949565646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:36.947162273 +0000 UTC m=+3790.332059878" watchObservedRunningTime="2026-11-01 21:05:36.949565646 +0000 UTC m=+3790.334463237"
Nov 01 21:05:47.842010 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:47.841978    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:05:48.008326 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:48.008291    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:05:51.629734 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:05:51.629693    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:05:53.009166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.009100    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" podUID="32d0e221-a880-4f55-ba0e-a745609a96b5" containerName="payload-processing" containerID="cri-o://c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986" gracePeriod=25
Nov 01 21:05:53.268053 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.267976    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:05:53.315386 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315347    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\") pod \"32d0e221-a880-4f55-ba0e-a745609a96b5\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") "
Nov 01 21:05:53.315634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315405    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\") pod \"32d0e221-a880-4f55-ba0e-a745609a96b5\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") "
Nov 01 21:05:53.315634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315432    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\") pod \"32d0e221-a880-4f55-ba0e-a745609a96b5\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") "
Nov 01 21:05:53.315634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315541    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\") pod \"32d0e221-a880-4f55-ba0e-a745609a96b5\" (UID: \"32d0e221-a880-4f55-ba0e-a745609a96b5\") "
Nov 01 21:05:53.315919 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315870    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume" pod "32d0e221-a880-4f55-ba0e-a745609a96b5" (UID: "32d0e221-a880-4f55-ba0e-a745609a96b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:05:53.316038 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.315958    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume" pod "32d0e221-a880-4f55-ba0e-a745609a96b5" (UID: "32d0e221-a880-4f55-ba0e-a745609a96b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:05:53.317834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.317805    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852" pod "32d0e221-a880-4f55-ba0e-a745609a96b5" (UID: "32d0e221-a880-4f55-ba0e-a745609a96b5"). InnerVolumeSpecName "kube-api-access-xz852". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:53.317834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.317813    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials" pod "32d0e221-a880-4f55-ba0e-a745609a96b5" (UID: "32d0e221-a880-4f55-ba0e-a745609a96b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:05:53.416508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.416456    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:53.416508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.416507    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz852\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-kube-api-access-xz852\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:53.416508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.416518    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32d0e221-a880-4f55-ba0e-a745609a96b5-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:53.416768 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.416527    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d0e221-a880-4f55-ba0e-a745609a96b5-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:05:53.929528 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.929475    5066 generic.go:356] "Generic (PLEG): container finished" podID="32d0e221-a880-4f55-ba0e-a745609a96b5" containerID="c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986" exitCode=0
Nov 01 21:05:53.929734 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.929551    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"
Nov 01 21:05:53.929734 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.929548    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" event={"ID":"32d0e221-a880-4f55-ba0e-a745609a96b5","Type":"ContainerDied","Data":"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"}
Nov 01 21:05:53.929734 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.929673    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7" event={"ID":"32d0e221-a880-4f55-ba0e-a745609a96b5","Type":"ContainerDied","Data":"9f2bb74a9bf635e67a186ea7fb7d61e5f3b8ceacfc87a8c2dd5e2d86375e9a40"}
Nov 01 21:05:53.929734 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.929695    5066 scope.go:122] "RemoveContainer" containerID="c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"
Nov 01 21:05:53.940695 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.940673    5066 scope.go:122] "RemoveContainer" containerID="c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"
Nov 01 21:05:53.941012 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:05:53.940990    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986\": container with ID starting with c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986 not found: ID does not exist" containerID="c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"
Nov 01 21:05:53.941070 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:53.941023    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986"} err="failed to get container status \"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986\": rpc error: code = NotFound desc = could not find container \"c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986\": container with ID starting with c5c7865b8a41499b3c7b13a4b52e7897045a0035f72946ef90807b436da16986 not found: ID does not exist"
Nov 01 21:05:54.093184 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:54.093145    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:05:54.142586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:54.142546    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449rhgl7"]
Nov 01 21:05:54.730783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:05:54.730737    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d0e221-a880-4f55-ba0e-a745609a96b5" path="/var/lib/kubelet/pods/32d0e221-a880-4f55-ba0e-a745609a96b5/volumes"
Nov 01 21:06:21.629201 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:06:21.629158    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:06:51.629020 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:06:51.628982    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:07:21.628284 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:07:21.628247    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:07:27.370794 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:07:27.370686    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:07:27.425311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:07:27.425285    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:07:51.628147 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:07:51.628066    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:08:21.628154 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:08:21.628118    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:08:32.131708 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:32.131664    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:08:32.132178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:32.131882    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy" containerID="cri-o://659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0" gracePeriod=30
Nov 01 21:08:37.374798 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.374770    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:08:37.475893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.475859    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.475920    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.475970    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.475996    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476231 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476063    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476231 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476097    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476231 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476140    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476231 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476172    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476293    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:37.476432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476395    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\") pod \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\" (UID: \"7b1747a8-0a08-47a3-9e19-04a6d38b9621\") "
Nov 01 21:08:37.476432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476410    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:37.476655 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476628    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:08:37.476655 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476644    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:37.476767 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476732    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:37.476952 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476930    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.477056 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476959    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.477056 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476973    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.477056 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.476987    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.477056 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.477001    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.478298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.478275    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:08:37.478298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.478287    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:08:37.478467 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.478449    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "kube-api-access-8b7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:08:37.478688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.478663    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo" pod "7b1747a8-0a08-47a3-9e19-04a6d38b9621" (UID: "7b1747a8-0a08-47a3-9e19-04a6d38b9621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:08:37.578573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.578528    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.578573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.578559    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.578573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.578568    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.578573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.578579    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7kd\" (UniqueName: \"kubernetes.io/projected/7b1747a8-0a08-47a3-9e19-04a6d38b9621-kube-api-access-8b7kd\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:08:37.838843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.838758    5066 generic.go:356] "Generic (PLEG): container finished" podID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerID="659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0" exitCode=0
Nov 01 21:08:37.838843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.838802    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"
Nov 01 21:08:37.839030 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.838850    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" event={"ID":"7b1747a8-0a08-47a3-9e19-04a6d38b9621","Type":"ContainerDied","Data":"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"}
Nov 01 21:08:37.839030 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.838901    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m" event={"ID":"7b1747a8-0a08-47a3-9e19-04a6d38b9621","Type":"ContainerDied","Data":"f48c789cf0d2d5978a0d4d196553214f19a4a8a8e9c2bab439bdd156f24b2b44"}
Nov 01 21:08:37.839030 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.838922    5066 scope.go:122] "RemoveContainer" containerID="659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"
Nov 01 21:08:37.851667 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.851636    5066 scope.go:122] "RemoveContainer" containerID="659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"
Nov 01 21:08:37.851991 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:08:37.851975    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0\": container with ID starting with 659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0 not found: ID does not exist" containerID="659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"
Nov 01 21:08:37.852068 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:37.851999    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0"} err="failed to get container status \"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0\": rpc error: code = NotFound desc = could not find container \"659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0\": container with ID starting with 659e9748c157418bb693577683f24ddffd0141c1d310d27d797e21a7127a15d0 not found: ID does not exist"
Nov 01 21:08:38.001679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:38.001634    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:08:38.051172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:38.051138    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dx22m"]
Nov 01 21:08:38.730245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:08:38.730209    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" path="/var/lib/kubelet/pods/7b1747a8-0a08-47a3-9e19-04a6d38b9621/volumes"
Nov 01 21:08:51.629244 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:08:51.629206    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:09:21.630139 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:09:21.630059    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:09:24.572027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:24.571984    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:09:29.573023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.572980    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" podUID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" containerName="payload-processing" containerID="cri-o://5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f" gracePeriod=25
Nov 01 21:09:29.832908 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.832837    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:09:29.925065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925024    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\") pod \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") "
Nov 01 21:09:29.925252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925083    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\") pod \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") "
Nov 01 21:09:29.925252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925134    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\") pod \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") "
Nov 01 21:09:29.925252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925156    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\") pod \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\" (UID: \"c7be7914-c4ad-46e9-9eaf-707bac1e5199\") "
Nov 01 21:09:29.925561 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925534    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume" pod "c7be7914-c4ad-46e9-9eaf-707bac1e5199" (UID: "c7be7914-c4ad-46e9-9eaf-707bac1e5199"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:09:29.925637 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.925589    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume" pod "c7be7914-c4ad-46e9-9eaf-707bac1e5199" (UID: "c7be7914-c4ad-46e9-9eaf-707bac1e5199"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:09:29.927348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.927323    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials" pod "c7be7914-c4ad-46e9-9eaf-707bac1e5199" (UID: "c7be7914-c4ad-46e9-9eaf-707bac1e5199"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:09:29.927348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:29.927332    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c" pod "c7be7914-c4ad-46e9-9eaf-707bac1e5199" (UID: "c7be7914-c4ad-46e9-9eaf-707bac1e5199"). InnerVolumeSpecName "kube-api-access-kcg5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:09:30.026993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.026953    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcg5c\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-kube-api-access-kcg5c\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:30.026993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.026986    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:30.026993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.026996    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7be7914-c4ad-46e9-9eaf-707bac1e5199-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:30.027242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.027006    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7be7914-c4ad-46e9-9eaf-707bac1e5199-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:09:30.131309 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.131226    5066 generic.go:356] "Generic (PLEG): container finished" podID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" containerID="5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f" exitCode=0
Nov 01 21:09:30.131309 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.131270    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" event={"ID":"c7be7914-c4ad-46e9-9eaf-707bac1e5199","Type":"ContainerDied","Data":"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"}
Nov 01 21:09:30.131309 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.131280    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"
Nov 01 21:09:30.131670 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.131312    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8" event={"ID":"c7be7914-c4ad-46e9-9eaf-707bac1e5199","Type":"ContainerDied","Data":"52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e"}
Nov 01 21:09:30.131670 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.131336    5066 scope.go:122] "RemoveContainer" containerID="5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"
Nov 01 21:09:30.147975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.147944    5066 scope.go:122] "RemoveContainer" containerID="5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"
Nov 01 21:09:30.148392 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:09:30.148366    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f\": container with ID starting with 5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f not found: ID does not exist" containerID="5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"
Nov 01 21:09:30.148483 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.148409    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f"} err="failed to get container status \"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f\": rpc error: code = NotFound desc = could not find container \"5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f\": container with ID starting with 5861eb8b7c09a5557c1ecf145b47cc47fe790f5be0409c1e5c0a2af9d522835f not found: ID does not exist"
Nov 01 21:09:30.170785 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:09:30.170735    5066 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be7914_c4ad_46e9_9eaf_707bac1e5199.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be7914_c4ad_46e9_9eaf_707bac1e5199.slice/crio-52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e\": RecentStats: unable to find data in memory cache]"
Nov 01 21:09:30.171002 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:09:30.170794    5066 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be7914_c4ad_46e9_9eaf_707bac1e5199.slice/crio-52715e6cbce8ba9173f390fe421b700e4605498686908eb4399a47937ca2972e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7be7914_c4ad_46e9_9eaf_707bac1e5199.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 21:09:30.293971 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.293932    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:09:30.296728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.296700    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75k5lx8"]
Nov 01 21:09:30.731048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:30.731015    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" path="/var/lib/kubelet/pods/c7be7914-c4ad-46e9-9eaf-707bac1e5199/volumes"
Nov 01 21:09:34.469995 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.469964    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:09:34.470552 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470534    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy"
Nov 01 21:09:34.470552 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470553    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy"
Nov 01 21:09:34.470658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470562    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" containerName="payload-processing"
Nov 01 21:09:34.470658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470567    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" containerName="payload-processing"
Nov 01 21:09:34.470658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470586    5066 cpu_manager.go:414] "containerMap: removing container" podUID="32d0e221-a880-4f55-ba0e-a745609a96b5" containerName="payload-processing"
Nov 01 21:09:34.470658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470592    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d0e221-a880-4f55-ba0e-a745609a96b5" containerName="payload-processing"
Nov 01 21:09:34.470817 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470672    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1747a8-0a08-47a3-9e19-04a6d38b9621" containerName="istio-proxy"
Nov 01 21:09:34.470817 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470679    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7be7914-c4ad-46e9-9eaf-707bac1e5199" containerName="payload-processing"
Nov 01 21:09:34.470817 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.470689    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d0e221-a880-4f55-ba0e-a745609a96b5" containerName="payload-processing"
Nov 01 21:09:34.473888 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.473867    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.575619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.575583    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.575801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.575629    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.575847 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.575817    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lz6n\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.575884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.575867    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576371    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a144a099ec31\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576397    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576401    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576434    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-f82019850b1a\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576440    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576447    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-9c12faaff243\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576471    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-f1b0d0f19d3d\""
Nov 01 21:09:34.576515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576489    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\""
Nov 01 21:09:34.576883 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.576605    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-zshnb\""
Nov 01 21:09:34.579335 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.579318    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:09:34.677574 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.677479    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.677754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.677609    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.677805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.677763    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lz6n\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.677860 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.677811    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.678363 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.678335    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.678506 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.678400    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.680628 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.680601    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.734096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.734025    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lz6n\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:34.789217 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:34.789180    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:35.167541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:35.167515    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:09:35.169699 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:09:35.169671    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e40375_406f_4362_b9ff_c365a907d466.slice/crio-7f7553e5dc8fc77ecbb596eba44aa9546a036ab7077e7e17f26271c759ab626b WatchSource:0}: Error finding container 7f7553e5dc8fc77ecbb596eba44aa9546a036ab7077e7e17f26271c759ab626b: Status 404 returned error can't find the container with id 7f7553e5dc8fc77ecbb596eba44aa9546a036ab7077e7e17f26271c759ab626b
Nov 01 21:09:36.168549 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.168483    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" event={"ID":"a0e40375-406f-4362-b9ff-c365a907d466","Type":"ContainerStarted","Data":"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"}
Nov 01 21:09:36.168972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.168577    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" event={"ID":"a0e40375-406f-4362-b9ff-c365a907d466","Type":"ContainerStarted","Data":"7f7553e5dc8fc77ecbb596eba44aa9546a036ab7077e7e17f26271c759ab626b"}
Nov 01 21:09:36.168972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.168793    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:36.281623 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.281567    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" podStartSLOduration=2.281547593 podStartE2EDuration="2.281547593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:36.281251334 +0000 UTC m=+4029.666148926" watchObservedRunningTime="2026-11-01 21:09:36.281547593 +0000 UTC m=+4029.666445183"
Nov 01 21:09:36.963439 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.963398    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:09:36.968124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:36.968094    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.021847 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.021809    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:09:37.068197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.068151    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\""
Nov 01 21:09:37.068197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.068158    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-896aa3063754\""
Nov 01 21:09:37.068197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.068199    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dd468d87ff44\""
Nov 01 21:09:37.068468 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.068218    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-nnqcq\""
Nov 01 21:09:37.068468 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.068151    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-14958f7d1994\""
Nov 01 21:09:37.070185 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.070165    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\""
Nov 01 21:09:37.070259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.070234    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\""
Nov 01 21:09:37.070259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.070246    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\""
Nov 01 21:09:37.070259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.070254    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-834a57f50f40\""
Nov 01 21:09:37.115753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.115722    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.115888 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.115792    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.115955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.115919    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.116027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.115972    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfwf\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.216944 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.216913    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.217436 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.216963    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.217436 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.216995    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfwf\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.217436 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.217136    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.217749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.217724    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.217949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.217926    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.219634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.219611    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.273320 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.273285    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfwf\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.287137 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.287110    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:37.667980 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:37.667949    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:09:37.670551 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:09:37.670517    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9299ecf_8d8d_4811_ba57_ff3725601217.slice/crio-b80c92ae96e953c3b4e7b16121cc2af70a68ef9d415044a7b72496bf6703b1c4 WatchSource:0}: Error finding container b80c92ae96e953c3b4e7b16121cc2af70a68ef9d415044a7b72496bf6703b1c4: Status 404 returned error can't find the container with id b80c92ae96e953c3b4e7b16121cc2af70a68ef9d415044a7b72496bf6703b1c4
Nov 01 21:09:38.182634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:38.182592    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" event={"ID":"d9299ecf-8d8d-4811-ba57-ff3725601217","Type":"ContainerStarted","Data":"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"}
Nov 01 21:09:38.182816 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:38.182656    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" event={"ID":"d9299ecf-8d8d-4811-ba57-ff3725601217","Type":"ContainerStarted","Data":"b80c92ae96e953c3b4e7b16121cc2af70a68ef9d415044a7b72496bf6703b1c4"}
Nov 01 21:09:38.182866 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:38.182825    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:38.294994 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:38.294882    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" podStartSLOduration=2.294868372 podStartE2EDuration="2.294868372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:09:38.294444458 +0000 UTC m=+4031.679342050" watchObservedRunningTime="2026-11-01 21:09:38.294868372 +0000 UTC m=+4031.679765964"
Nov 01 21:09:47.178155 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:47.178122    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:09:49.191483 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:09:49.191454    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:09:51.628418 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:09:51.628378    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:10:21.628547 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:10:21.628511    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:10:51.627753 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:10:51.627710    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:11:03.245744 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:03.245702    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:11:03.246290 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:03.245921    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy" containerID="cri-o://3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28" gracePeriod=30
Nov 01 21:11:08.504940 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.504910    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:11:08.653318 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653256    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653345    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653398    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653428    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653460    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653565    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:11:08.653673 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653593    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:11:08.653673 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653616    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653788 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653671    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653788 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653714    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653788 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653743    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\") pod \"b4047a62-1230-47bb-9b80-7ed53390deb2\" (UID: \"b4047a62-1230-47bb-9b80-7ed53390deb2\") "
Nov 01 21:11:08.653947 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653898    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:11:08.654014 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.653980    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:11:08.654121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654102    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:11:08.654356 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654335    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.654420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654362    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.654420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654377    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4047a62-1230-47bb-9b80-7ed53390deb2-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.654420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654392    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.654420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.654407    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.655815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.655787    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:11:08.655928 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.655814    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:11:08.656119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.656092    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:11:08.656119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.656103    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b" pod "b4047a62-1230-47bb-9b80-7ed53390deb2" (UID: "b4047a62-1230-47bb-9b80-7ed53390deb2"). InnerVolumeSpecName "kube-api-access-rn25b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:11:08.701899 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.701864    5066 generic.go:356] "Generic (PLEG): container finished" podID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerID="3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28" exitCode=0
Nov 01 21:11:08.702023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.701910    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" event={"ID":"b4047a62-1230-47bb-9b80-7ed53390deb2","Type":"ContainerDied","Data":"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"}
Nov 01 21:11:08.702023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.701952    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz" event={"ID":"b4047a62-1230-47bb-9b80-7ed53390deb2","Type":"ContainerDied","Data":"06849851ab37a22b91e7aa29a24795db2349fd0d1eb09106239c380fd47ac116"}
Nov 01 21:11:08.702023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.701974    5066 scope.go:122] "RemoveContainer" containerID="3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"
Nov 01 21:11:08.702134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.702096    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"
Nov 01 21:11:08.712459 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.712442    5066 scope.go:122] "RemoveContainer" containerID="3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"
Nov 01 21:11:08.712752 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:11:08.712733    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28\": container with ID starting with 3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28 not found: ID does not exist" containerID="3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"
Nov 01 21:11:08.712803 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.712758    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28"} err="failed to get container status \"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28\": rpc error: code = NotFound desc = could not find container \"3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28\": container with ID starting with 3f9be70976144b8105c122354ddbf7de27e1fa0258b41969c913cde6e676bc28 not found: ID does not exist"
Nov 01 21:11:08.755045 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.755019    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.755045 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.755041    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn25b\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-kube-api-access-rn25b\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.755188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.755052    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.755188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.755062    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4047a62-1230-47bb-9b80-7ed53390deb2-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:11:08.866051 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.866021    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:11:08.869663 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:08.869638    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cr2pz"]
Nov 01 21:11:10.730628 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:11:10.730593    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" path="/var/lib/kubelet/pods/b4047a62-1230-47bb-9b80-7ed53390deb2/volumes"
Nov 01 21:11:21.629334 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:11:21.629295    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:11:51.628848 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:11:51.628811    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:12:21.627925 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:12:21.627845    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:12:27.408255 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:12:27.408142    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:12:27.463125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:12:27.463100    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:12:51.627186 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:12:51.627146    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:13:21.627377 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:13:21.627333    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:13:51.627113 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:13:51.627073    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:14:21.627337 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:21.627302    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:14:33.825709 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:33.825672    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:33.825709 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:33.825701    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:33.825709 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:33.825715    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:33.826442 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:33.825766    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:34.325752172 +0000 UTC m=+4327.710649753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:34.330790 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:34.330756    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-9c12faaff243: secret "agc-cred-9c12faaff243" not found
Nov 01 21:14:34.330790 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:34.330788    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:34.330790 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:34.330798    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:34.331023 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:34.330807    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:34.331023 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:34.330856    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:35.330843894 +0000 UTC m=+4328.715741475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:35.343019 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:35.342986    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-9c12faaff243: secret "agc-cred-9c12faaff243" not found
Nov 01 21:14:35.343019 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:35.343019    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:35.343019 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:35.343029    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:35.343598 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:35.343040    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:35.343598 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:35.343118    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:37.343096192 +0000 UTC m=+4330.727993765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:37.362809 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:37.362767    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-9c12faaff243: secret "agc-cred-9c12faaff243" not found
Nov 01 21:14:37.362809 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:37.362805    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:37.362809 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:37.362814    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:37.363370 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:37.362825    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:37.363370 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:37.362889    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:41.362868959 +0000 UTC m=+4334.747766543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:38.880116 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:38.879800    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:38.880116 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:38.879832    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:38.880116 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:38.879843    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:38.880116 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:38.879861    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:38.880116 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:38.879935    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:39.379915219 +0000 UTC m=+4332.764812799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:39.385181 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:39.385141    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:39.385181 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:39.385174    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:39.385181 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:39.385183    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:39.385521 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:39.385199    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:39.385521 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:39.385272    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:40.385253861 +0000 UTC m=+4333.770151431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:40.394898 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:40.394860    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:40.394898 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:40.394888    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:40.394898 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:40.394896    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:40.395339 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:40.394909    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:40.395339 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:40.394967    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:42.394951141 +0000 UTC m=+4335.779848729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:41.404895 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:41.404861    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-9c12faaff243: secret "agc-cred-9c12faaff243" not found
Nov 01 21:14:41.404895 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:41.404894    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:41.404895 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:41.404903    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:41.405336 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:41.404911    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:41.405336 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:41.404963    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:49.404949422 +0000 UTC m=+4342.789847006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:42.415134 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:42.415105    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:42.415134 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:42.415133    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:42.415134 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:42.415139    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:42.415609 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:42.415156    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:42.415609 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:42.415209    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:46.415195597 +0000 UTC m=+4339.800093169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:46.457218 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:46.457189    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:46.457218 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:46.457215    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:46.457218 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:46.457222    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:46.457781 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:46.457234    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:46.457781 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:46.457291    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:14:54.457278905 +0000 UTC m=+4347.842176486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:49.483835 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:49.483802    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-9c12faaff243: secret "agc-cred-9c12faaff243" not found
Nov 01 21:14:49.484221 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:49.483844    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f1b0d0f19d3d: secret "agc-cred-f1b0d0f19d3d" not found
Nov 01 21:14:49.484221 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:49.483853    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-f82019850b1a: secret "agc-cred-f82019850b1a" not found
Nov 01 21:14:49.484221 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:49.483866    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88: [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:49.484221 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:49.483937    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials podName:a0e40375-406f-4362-b9ff-c365a907d466 nodeName:}" failed. No retries permitted until 2026-11-01 21:15:05.483920596 +0000 UTC m=+4358.868818171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" (UID: "a0e40375-406f-4362-b9ff-c365a907d466") : [secret "agc-cred-9c12faaff243" not found, secret "agc-cred-f1b0d0f19d3d" not found, secret "agc-cred-f82019850b1a" not found]
Nov 01 21:14:49.869915 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:49.869876    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:14:49.870149 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:49.870124    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" containerID="cri-o://6afdc3ff99bd82a06f86f35fbafec5bb236056fd07cc4ea327b10329a58e1c34" gracePeriod=30
Nov 01 21:14:51.627471 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:51.627437    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:14:54.529950 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:54.529912    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-14958f7d1994: secret "agc-cred-14958f7d1994" not found
Nov 01 21:14:54.529950 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:54.529943    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-834a57f50f40: secret "agc-cred-834a57f50f40" not found
Nov 01 21:14:54.529950 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:54.529952    5066 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-896aa3063754: secret "agc-cred-896aa3063754" not found
Nov 01 21:14:54.530540 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:54.529970    5066 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb: [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:54.530540 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:14:54.530049    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials podName:d9299ecf-8d8d-4811-ba57-ff3725601217 nodeName:}" failed. No retries permitted until 2026-11-01 21:15:10.530027456 +0000 UTC m=+4363.914925028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217") : [secret "agc-cred-14958f7d1994" not found, secret "agc-cred-834a57f50f40" not found, secret "agc-cred-896aa3063754" not found]
Nov 01 21:14:54.558029 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:54.557990    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5s667 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:14:54.558208 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:54.558067    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:14:54.967620 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:54.967585    5066 generic.go:356] "Generic (PLEG): container finished" podID="be8f9686-1afa-4299-9583-d92a035a3c67" containerID="6afdc3ff99bd82a06f86f35fbafec5bb236056fd07cc4ea327b10329a58e1c34" exitCode=0
Nov 01 21:14:54.967801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:54.967665    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" event={"ID":"be8f9686-1afa-4299-9583-d92a035a3c67","Type":"ContainerDied","Data":"6afdc3ff99bd82a06f86f35fbafec5bb236056fd07cc4ea327b10329a58e1c34"}
Nov 01 21:14:55.523360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.523328    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:14:55.538564 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538538    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538575    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538612    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538641    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538723    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538789    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538857    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538875    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.538936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538914    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.538979    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539007    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539077    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\") pod \"be8f9686-1afa-4299-9583-d92a035a3c67\" (UID: \"be8f9686-1afa-4299-9583-d92a035a3c67\") "
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539099    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539119    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:14:55.539424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539416    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.539782 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539437    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.539782 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539453    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.539782 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539465    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.539782 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.539478    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f9686-1afa-4299-9583-d92a035a3c67-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.541013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.540989    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:14:55.541096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.541009    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:14:55.541096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.541017    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:55.541096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.541075    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8" pod "be8f9686-1afa-4299-9583-d92a035a3c67" (UID: "be8f9686-1afa-4299-9583-d92a035a3c67"). InnerVolumeSpecName "kube-api-access-j4wv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:14:55.641193 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.641105    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wv8\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-kube-api-access-j4wv8\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.641412 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.641387    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f9686-1afa-4299-9583-d92a035a3c67-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.641558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.641415    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f9686-1afa-4299-9583-d92a035a3c67-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.641558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.641448    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f9686-1afa-4299-9583-d92a035a3c67-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:14:55.974695 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.974662    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"
Nov 01 21:14:55.974882 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.974666    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667" event={"ID":"be8f9686-1afa-4299-9583-d92a035a3c67","Type":"ContainerDied","Data":"b3d3f53a3645ba6d69f5b39d822634b7ebe7a80d39393be7c12585cff1567bf9"}
Nov 01 21:14:55.974882 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:55.974790    5066 scope.go:122] "RemoveContainer" containerID="6afdc3ff99bd82a06f86f35fbafec5bb236056fd07cc4ea327b10329a58e1c34"
Nov 01 21:14:56.138315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:56.138271    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:14:56.143450 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:56.143420    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5s667"]
Nov 01 21:14:56.730756 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:56.730719    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" path="/var/lib/kubelet/pods/be8f9686-1afa-4299-9583-d92a035a3c67/volumes"
Nov 01 21:14:59.949980 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:14:59.949935    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:15:02.614175 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:02.614126    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:15:04.950845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:04.950808    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" podUID="a0e40375-406f-4362-b9ff-c365a907d466" containerName="payload-processing" containerID="cri-o://2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a" gracePeriod=25
Nov 01 21:15:05.199869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.199844    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:15:05.242837 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.242807    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\") pod \"a0e40375-406f-4362-b9ff-c365a907d466\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") "
Nov 01 21:15:05.242992 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.242869    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\") pod \"a0e40375-406f-4362-b9ff-c365a907d466\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") "
Nov 01 21:15:05.243035 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.242995    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\") pod \"a0e40375-406f-4362-b9ff-c365a907d466\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") "
Nov 01 21:15:05.243078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.243056    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\") pod \"a0e40375-406f-4362-b9ff-c365a907d466\" (UID: \"a0e40375-406f-4362-b9ff-c365a907d466\") "
Nov 01 21:15:05.243302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.243207    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume" pod "a0e40375-406f-4362-b9ff-c365a907d466" (UID: "a0e40375-406f-4362-b9ff-c365a907d466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:05.243426 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.243324    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume" pod "a0e40375-406f-4362-b9ff-c365a907d466" (UID: "a0e40375-406f-4362-b9ff-c365a907d466"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:05.243426 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.243392    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:05.245045 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.245021    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials" pod "a0e40375-406f-4362-b9ff-c365a907d466" (UID: "a0e40375-406f-4362-b9ff-c365a907d466"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:05.245195 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.245103    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n" pod "a0e40375-406f-4362-b9ff-c365a907d466" (UID: "a0e40375-406f-4362-b9ff-c365a907d466"). InnerVolumeSpecName "kube-api-access-8lz6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:05.344602 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.344571    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lz6n\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-kube-api-access-8lz6n\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:05.344602 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.344598    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e40375-406f-4362-b9ff-c365a907d466-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:05.344602 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:05.344607    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e40375-406f-4362-b9ff-c365a907d466-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:06.032859 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.032820    5066 generic.go:356] "Generic (PLEG): container finished" podID="a0e40375-406f-4362-b9ff-c365a907d466" containerID="2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a" exitCode=0
Nov 01 21:15:06.033370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.032884    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"
Nov 01 21:15:06.033370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.032910    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" event={"ID":"a0e40375-406f-4362-b9ff-c365a907d466","Type":"ContainerDied","Data":"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"}
Nov 01 21:15:06.033370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.032968    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88" event={"ID":"a0e40375-406f-4362-b9ff-c365a907d466","Type":"ContainerDied","Data":"7f7553e5dc8fc77ecbb596eba44aa9546a036ab7077e7e17f26271c759ab626b"}
Nov 01 21:15:06.033370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.032989    5066 scope.go:122] "RemoveContainer" containerID="2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"
Nov 01 21:15:06.043476 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.043451    5066 scope.go:122] "RemoveContainer" containerID="2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"
Nov 01 21:15:06.043790 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:06.043766    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a\": container with ID starting with 2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a not found: ID does not exist" containerID="2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"
Nov 01 21:15:06.043839 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.043805    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a"} err="failed to get container status \"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a\": rpc error: code = NotFound desc = could not find container \"2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a\": container with ID starting with 2210a075adcf3009d5cdb3b4bd490f81d83f81bf1b5d463db26443f2c9b51a4a not found: ID does not exist"
Nov 01 21:15:06.199119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.199082    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:15:06.248364 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.248327    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-7bb4bbq88"]
Nov 01 21:15:06.730834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:06.730799    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e40375-406f-4362-b9ff-c365a907d466" path="/var/lib/kubelet/pods/a0e40375-406f-4362-b9ff-c365a907d466/volumes"
Nov 01 21:15:07.615550 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.615477    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" podUID="d9299ecf-8d8d-4811-ba57-ff3725601217" containerName="payload-processing" containerID="cri-o://6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c" gracePeriod=25
Nov 01 21:15:07.863068 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.863043    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:15:07.971010 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.970983    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\") pod \"d9299ecf-8d8d-4811-ba57-ff3725601217\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") "
Nov 01 21:15:07.971209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.971072    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\") pod \"d9299ecf-8d8d-4811-ba57-ff3725601217\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") "
Nov 01 21:15:07.971209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.971107    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\") pod \"d9299ecf-8d8d-4811-ba57-ff3725601217\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") "
Nov 01 21:15:07.971209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.971126    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\") pod \"d9299ecf-8d8d-4811-ba57-ff3725601217\" (UID: \"d9299ecf-8d8d-4811-ba57-ff3725601217\") "
Nov 01 21:15:07.971788 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.971753    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume" pod "d9299ecf-8d8d-4811-ba57-ff3725601217" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:07.971908 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.971815    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume" pod "d9299ecf-8d8d-4811-ba57-ff3725601217" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:07.973751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.973718    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials" pod "d9299ecf-8d8d-4811-ba57-ff3725601217" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:07.973840 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:07.973829    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf" pod "d9299ecf-8d8d-4811-ba57-ff3725601217" (UID: "d9299ecf-8d8d-4811-ba57-ff3725601217"). InnerVolumeSpecName "kube-api-access-bqfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:08.047263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.047229    5066 generic.go:356] "Generic (PLEG): container finished" podID="d9299ecf-8d8d-4811-ba57-ff3725601217" containerID="6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c" exitCode=0
Nov 01 21:15:08.047434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.047280    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" event={"ID":"d9299ecf-8d8d-4811-ba57-ff3725601217","Type":"ContainerDied","Data":"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"}
Nov 01 21:15:08.047434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.047321    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb" event={"ID":"d9299ecf-8d8d-4811-ba57-ff3725601217","Type":"ContainerDied","Data":"b80c92ae96e953c3b4e7b16121cc2af70a68ef9d415044a7b72496bf6703b1c4"}
Nov 01 21:15:08.047434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.047341    5066 scope.go:122] "RemoveContainer" containerID="6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"
Nov 01 21:15:08.047434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.047283    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"
Nov 01 21:15:08.057646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.057623    5066 scope.go:122] "RemoveContainer" containerID="6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"
Nov 01 21:15:08.057931 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:08.057911    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c\": container with ID starting with 6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c not found: ID does not exist" containerID="6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"
Nov 01 21:15:08.058001 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.057938    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c"} err="failed to get container status \"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c\": rpc error: code = NotFound desc = could not find container \"6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c\": container with ID starting with 6783e1800a61fc00d626790bc7d22e4c77c2cc3427596ea14fc5995cc494694c not found: ID does not exist"
Nov 01 21:15:08.072586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.072564    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:08.072586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.072584    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:08.072725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.072595    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9299ecf-8d8d-4811-ba57-ff3725601217-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:08.072725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.072604    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqfwf\" (UniqueName: \"kubernetes.io/projected/d9299ecf-8d8d-4811-ba57-ff3725601217-kube-api-access-bqfwf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:08.209987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.209951    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:15:08.212815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.212789    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-76ccgkqpb"]
Nov 01 21:15:08.729720 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:08.729689    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9299ecf-8d8d-4811-ba57-ff3725601217" path="/var/lib/kubelet/pods/d9299ecf-8d8d-4811-ba57-ff3725601217/volumes"
Nov 01 21:15:12.268288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:12.268251    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 21:15:12.376187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:12.376150    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 21:15:17.268993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:17.268949    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing" containerID="cri-o://c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591" gracePeriod=25
Nov 01 21:15:17.376807 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:17.376753    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing" containerID="cri-o://e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72" gracePeriod=25
Nov 01 21:15:21.627308 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:21.627271    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:15:22.499334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.499299    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 21:15:22.499547 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.499523    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" podUID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" containerName="maas-api" containerID="cri-o://b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18" gracePeriod=30
Nov 01 21:15:22.757989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.757917    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 21:15:22.915875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.915843    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\" (UniqueName: \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\") pod \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") "
Nov 01 21:15:22.915875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.915880    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\") pod \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") "
Nov 01 21:15:22.916142 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.915911    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\") pod \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") "
Nov 01 21:15:22.916142 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.915984    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") pod \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\" (UID: \"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb\") "
Nov 01 21:15:22.916472 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.916438    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca" pod "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" (UID: "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:22.918041 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.918006    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls" pod "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" (UID: "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:22.918132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.918081    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs" pod "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" (UID: "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:22.918132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:22.918107    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv" pod "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" (UID: "c5d4897f-7ea5-4cf2-a861-6a35f1979dfb"). InnerVolumeSpecName "kube-api-access-w2qxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:23.017420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.017329    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qxv\" (UniqueName: \"kubernetes.io/projected/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-kube-api-access-w2qxv\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:23.017420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.017370    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:23.017420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.017381    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:23.017420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.017390    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:23.143859 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.143826    5066 generic.go:356] "Generic (PLEG): container finished" podID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" containerID="b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18" exitCode=0
Nov 01 21:15:23.144022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.143879    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" event={"ID":"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb","Type":"ContainerDied","Data":"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"}
Nov 01 21:15:23.144022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.143888    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"
Nov 01 21:15:23.144022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.143921    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8" event={"ID":"c5d4897f-7ea5-4cf2-a861-6a35f1979dfb","Type":"ContainerDied","Data":"49b29844e33f6b97d36dc19c8c75cd64652ee208ab0011d03f087923c1b80170"}
Nov 01 21:15:23.144022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.143943    5066 scope.go:122] "RemoveContainer" containerID="b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"
Nov 01 21:15:23.157804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.157777    5066 scope.go:122] "RemoveContainer" containerID="b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"
Nov 01 21:15:23.158097 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:23.158074    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18\": container with ID starting with b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18 not found: ID does not exist" containerID="b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"
Nov 01 21:15:23.158173 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.158105    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18"} err="failed to get container status \"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18\": rpc error: code = NotFound desc = could not find container \"b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18\": container with ID starting with b54acae4d55eb5a17d683b49e024cf2d5f1843c46bd5df071e98896810e47a18 not found: ID does not exist"
Nov 01 21:15:23.306067 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.305979    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 21:15:23.311519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:23.311481    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99d504-5969b94cc9-qhwx8"]
Nov 01 21:15:24.729930 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:24.729897    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" path="/var/lib/kubelet/pods/c5d4897f-7ea5-4cf2-a861-6a35f1979dfb/volumes"
Nov 01 21:15:26.053714 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:26.053679    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:26.054165 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:26.053678    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:34.649837 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:34.649803    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 21:15:36.053786 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:36.053748    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:36.053786 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:36.053768    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:37.522508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.522459    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 21:15:37.555157 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.555124    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\" (UniqueName: \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\") pod \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") "
Nov 01 21:15:37.555311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.555204    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\") pod \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\" (UID: \"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c\") "
Nov 01 21:15:37.555612 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.555584    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume" pod "fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" (UID: "fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:37.557711 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.557678    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9" pod "fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" (UID: "fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c"). InnerVolumeSpecName "kube-api-access-l4rx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:37.607845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.607819    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 21:15:37.656661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.656632    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\") pod \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") "
Nov 01 21:15:37.656831 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.656726    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\" (UniqueName: \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\") pod \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\" (UID: \"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511\") "
Nov 01 21:15:37.656999 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.656976    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume" pod "2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" (UID: "2cccca0e-4a10-4c39-a2c3-7af0ca2e7511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:37.657044 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.657014    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rx9\" (UniqueName: \"kubernetes.io/projected/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-kube-api-access-l4rx9\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:37.657044 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.657036    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:37.658924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.658900    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b" pod "2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" (UID: "2cccca0e-4a10-4c39-a2c3-7af0ca2e7511"). InnerVolumeSpecName "kube-api-access-tfx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:37.758327 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.758291    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:37.758327 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:37.758320    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfx9b\" (UniqueName: \"kubernetes.io/projected/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511-kube-api-access-tfx9b\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:38.224558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.224526    5066 generic.go:356] "Generic (PLEG): container finished" podID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerID="c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591" exitCode=0
Nov 01 21:15:38.224753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.224570    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"
Nov 01 21:15:38.224753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.224603    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" event={"ID":"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c","Type":"ContainerDied","Data":"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"}
Nov 01 21:15:38.224753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.224662    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8" event={"ID":"fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c","Type":"ContainerDied","Data":"798dc8c3e367f766cd3a379437803eaa16f3c9f7aba888b7928e8ef1c64492f0"}
Nov 01 21:15:38.224753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.224690    5066 scope.go:122] "RemoveContainer" containerID="c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"
Nov 01 21:15:38.226172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.226137    5066 generic.go:356] "Generic (PLEG): container finished" podID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerID="e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72" exitCode=0
Nov 01 21:15:38.226172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.226164    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" event={"ID":"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511","Type":"ContainerDied","Data":"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"}
Nov 01 21:15:38.226351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.226205    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm" event={"ID":"2cccca0e-4a10-4c39-a2c3-7af0ca2e7511","Type":"ContainerDied","Data":"38c94dfd8d664490fdf0afcdc161b3d4461ac126b61a1a3933c7f4ee83efa590"}
Nov 01 21:15:38.226351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.226204    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"
Nov 01 21:15:38.234326 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.234309    5066 scope.go:122] "RemoveContainer" containerID="c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"
Nov 01 21:15:38.234604 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:38.234574    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591\": container with ID starting with c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591 not found: ID does not exist" containerID="c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"
Nov 01 21:15:38.234726 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.234610    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591"} err="failed to get container status \"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591\": rpc error: code = NotFound desc = could not find container \"c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591\": container with ID starting with c9ace89b13eac1d1f4aade11483d4bc0f240408ef358f1547c7bc75db2277591 not found: ID does not exist"
Nov 01 21:15:38.234726 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.234632    5066 scope.go:122] "RemoveContainer" containerID="e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"
Nov 01 21:15:38.242789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.242770    5066 scope.go:122] "RemoveContainer" containerID="e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"
Nov 01 21:15:38.243067 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:38.243045    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72\": container with ID starting with e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72 not found: ID does not exist" containerID="e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"
Nov 01 21:15:38.243144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.243075    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72"} err="failed to get container status \"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72\": rpc error: code = NotFound desc = could not find container \"e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72\": container with ID starting with e1d679034d3f10d5a3d97a83ecaa82d53643bfdb99426830a11fcb6671584c72 not found: ID does not exist"
Nov 01 21:15:38.388733 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.388693    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 21:15:38.438098 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.438062    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99d504-74b9c6f475-fb9lm"]
Nov 01 21:15:38.500900 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.500808    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 21:15:38.503025 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.502999    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99d504-7889fb5dfc-sr2f8"]
Nov 01 21:15:38.729754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.729718    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" path="/var/lib/kubelet/pods/2cccca0e-4a10-4c39-a2c3-7af0ca2e7511/volumes"
Nov 01 21:15:38.730133 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:38.730103    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" path="/var/lib/kubelet/pods/fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c/volumes"
Nov 01 21:15:39.650903 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:39.650859    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing" containerID="cri-o://b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e" gracePeriod=25
Nov 01 21:15:43.010797 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.010755    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 21:15:43.011243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.010995    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" podUID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" containerName="maas-api" containerID="cri-o://efcbc8a91630a1ab120ed7909f7231732cc561868610021e5237f6c48c3c9df7" gracePeriod=30
Nov 01 21:15:43.254770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.254740    5066 generic.go:356] "Generic (PLEG): container finished" podID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" containerID="efcbc8a91630a1ab120ed7909f7231732cc561868610021e5237f6c48c3c9df7" exitCode=0
Nov 01 21:15:43.254921 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.254814    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" event={"ID":"55054cc1-2b4a-452e-84e7-ff27b6e9cc71","Type":"ContainerDied","Data":"efcbc8a91630a1ab120ed7909f7231732cc561868610021e5237f6c48c3c9df7"}
Nov 01 21:15:43.254921 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.254866    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62" event={"ID":"55054cc1-2b4a-452e-84e7-ff27b6e9cc71","Type":"ContainerDied","Data":"862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66"}
Nov 01 21:15:43.254921 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.254888    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862ed5314173d7bb14436340a7b1c0157e901b4920afd6379de1678a4cbefc66"
Nov 01 21:15:43.268740 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.268687    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 21:15:43.313919 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.313882    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\" (UniqueName: \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\") pod \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") "
Nov 01 21:15:43.314119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.313948    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\") pod \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") "
Nov 01 21:15:43.314119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.313999    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\") pod \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") "
Nov 01 21:15:43.314119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.314033    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") pod \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\" (UID: \"55054cc1-2b4a-452e-84e7-ff27b6e9cc71\") "
Nov 01 21:15:43.314350 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.314322    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca" pod "55054cc1-2b4a-452e-84e7-ff27b6e9cc71" (UID: "55054cc1-2b4a-452e-84e7-ff27b6e9cc71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:43.316092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.316064    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff" pod "55054cc1-2b4a-452e-84e7-ff27b6e9cc71" (UID: "55054cc1-2b4a-452e-84e7-ff27b6e9cc71"). InnerVolumeSpecName "kube-api-access-7psff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:43.316222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.316118    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs" pod "55054cc1-2b4a-452e-84e7-ff27b6e9cc71" (UID: "55054cc1-2b4a-452e-84e7-ff27b6e9cc71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:43.316222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.316133    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls" pod "55054cc1-2b4a-452e-84e7-ff27b6e9cc71" (UID: "55054cc1-2b4a-452e-84e7-ff27b6e9cc71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:15:43.414954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.414921    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:43.414954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.414950    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:43.414954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.414960    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:43.415182 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:43.414969    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7psff\" (UniqueName: \"kubernetes.io/projected/55054cc1-2b4a-452e-84e7-ff27b6e9cc71-kube-api-access-7psff\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:44.260651 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:44.260617    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"
Nov 01 21:15:44.423053 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:44.423013    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 21:15:44.472906 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:44.472870    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6a675a-767f8f57-prf62"]
Nov 01 21:15:44.731012 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:44.730967    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" path="/var/lib/kubelet/pods/55054cc1-2b4a-452e-84e7-ff27b6e9cc71/volumes"
Nov 01 21:15:46.429798 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:46.429762    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 21:15:46.430262 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:46.429980    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" containerID="cri-o://bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072" gracePeriod=30
Nov 01 21:15:47.421509 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:47.421464    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:47.842352 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:47.842313    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4pll5_631dad46-02be-4fba-a072-43be25b7a0e5/manager/0.log"
Nov 01 21:15:49.822067 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:49.822021    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mjh7s_0da5efcb-cdd3-48d5-813d-17bc1d59717c/manager/0.log"
Nov 01 21:15:50.226997 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:50.226956    5066 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:15:50.227166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:50.227046    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:15:51.636524 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:51.636238    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:15:51.684070 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.684042    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 21:15:51.799808 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799773    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.799808 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799815    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799878    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799912    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799937    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.799960    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800013    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800059    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800123    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\") pod \"caff215e-7392-40d2-b79e-9cb3227d92e2\" (UID: \"caff215e-7392-40d2-b79e-9cb3227d92e2\") "
Nov 01 21:15:51.800395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800241    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:51.800395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800253    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:51.800395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800267    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:51.800643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800458    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.800643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800478    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.800643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800506    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.800643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800469    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:51.800643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.800632    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:51.802541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.802520    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:51.802697 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.802664    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "kube-api-access-jcghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:15:51.802806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.802749    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:15:51.802806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.802756    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy" pod "caff215e-7392-40d2-b79e-9cb3227d92e2" (UID: "caff215e-7392-40d2-b79e-9cb3227d92e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:15:51.902019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.901980    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcghk\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-kube-api-access-jcghk\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.902019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.902015    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.902019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.902025    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caff215e-7392-40d2-b79e-9cb3227d92e2-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.902262 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.902034    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.902262 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.902043    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caff215e-7392-40d2-b79e-9cb3227d92e2-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:51.902262 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:51.902052    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caff215e-7392-40d2-b79e-9cb3227d92e2-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:15:52.142124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.142031    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cmbpq_be0efee3-c422-4d59-a0b7-67c656df942c/manager/0.log"
Nov 01 21:15:52.308445 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.308412    5066 generic.go:356] "Generic (PLEG): container finished" podID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerID="bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072" exitCode=0
Nov 01 21:15:52.308640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.308460    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"
Nov 01 21:15:52.308640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.308463    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" event={"ID":"caff215e-7392-40d2-b79e-9cb3227d92e2","Type":"ContainerDied","Data":"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"}
Nov 01 21:15:52.308640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.308569    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955" event={"ID":"caff215e-7392-40d2-b79e-9cb3227d92e2","Type":"ContainerDied","Data":"016b6e2cedf2af71d002769cae29538d283b588058db3565e157b637379e7e32"}
Nov 01 21:15:52.308640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.308599    5066 scope.go:122] "RemoveContainer" containerID="bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"
Nov 01 21:15:52.318882 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.318861    5066 scope.go:122] "RemoveContainer" containerID="bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"
Nov 01 21:15:52.319168 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:15:52.319153    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072\": container with ID starting with bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072 not found: ID does not exist" containerID="bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"
Nov 01 21:15:52.319207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.319176    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072"} err="failed to get container status \"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072\": rpc error: code = NotFound desc = could not find container \"bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072\": container with ID starting with bd318b843c81d3f76126e7f17344f6edc3719f763bcd8e8a99fe2fec52c78072 not found: ID does not exist"
Nov 01 21:15:52.521012 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.520974    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 21:15:52.528912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.528883    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-6q955"]
Nov 01 21:15:52.731409 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:52.731366    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" path="/var/lib/kubelet/pods/caff215e-7392-40d2-b79e-9cb3227d92e2/volumes"
Nov 01 21:15:57.422616 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:57.422569    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:15:59.903606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:59.903578    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 21:15:59.978537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:59.978508    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\" (UniqueName: \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\") pod \"292b3de2-9c16-4c09-892c-2c72992231b4\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") "
Nov 01 21:15:59.978697 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:59.978601    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\") pod \"292b3de2-9c16-4c09-892c-2c72992231b4\" (UID: \"292b3de2-9c16-4c09-892c-2c72992231b4\") "
Nov 01 21:15:59.978966 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:59.978933    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume" pod "292b3de2-9c16-4c09-892c-2c72992231b4" (UID: "292b3de2-9c16-4c09-892c-2c72992231b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:15:59.980560 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:15:59.980540    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz" pod "292b3de2-9c16-4c09-892c-2c72992231b4" (UID: "292b3de2-9c16-4c09-892c-2c72992231b4"). InnerVolumeSpecName "kube-api-access-tpdnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:00.080284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.080241    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292b3de2-9c16-4c09-892c-2c72992231b4-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.080284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.080278    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdnz\" (UniqueName: \"kubernetes.io/projected/292b3de2-9c16-4c09-892c-2c72992231b4-kube-api-access-tpdnz\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:00.353125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.353031    5066 generic.go:356] "Generic (PLEG): container finished" podID="292b3de2-9c16-4c09-892c-2c72992231b4" containerID="b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e" exitCode=0
Nov 01 21:16:00.353125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.353082    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"
Nov 01 21:16:00.353125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.353116    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" event={"ID":"292b3de2-9c16-4c09-892c-2c72992231b4","Type":"ContainerDied","Data":"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"}
Nov 01 21:16:00.353379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.353171    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt" event={"ID":"292b3de2-9c16-4c09-892c-2c72992231b4","Type":"ContainerDied","Data":"d041298d4cf54f36d9a6e8d8eb5642ad7c7abffeaa4dde9a3284399ccfba0f59"}
Nov 01 21:16:00.353379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.353196    5066 scope.go:122] "RemoveContainer" containerID="b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"
Nov 01 21:16:00.370369 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.370334    5066 scope.go:122] "RemoveContainer" containerID="b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"
Nov 01 21:16:00.370769 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:00.370744    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e\": container with ID starting with b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e not found: ID does not exist" containerID="b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"
Nov 01 21:16:00.370846 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.370776    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e"} err="failed to get container status \"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e\": rpc error: code = NotFound desc = could not find container \"b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e\": container with ID starting with b28ab59f0b9a0f1224b7f6202b557cc833e6f64f44114585dc34aefed352ca2e not found: ID does not exist"
Nov 01 21:16:00.515520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.515455    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 21:16:00.518060 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.518027    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6a675a-6d59645857-k8szt"]
Nov 01 21:16:00.729631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:00.729600    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" path="/var/lib/kubelet/pods/292b3de2-9c16-4c09-892c-2c72992231b4/volumes"
Nov 01 21:16:02.259990 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.259951    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260352    5066 cpu_manager.go:414] "containerMap: removing container" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy"
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260363    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy"
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260374    5066 cpu_manager.go:414] "containerMap: removing container" podUID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" containerName="maas-api"
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260379    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" containerName="maas-api"
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260385    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" containerName="maas-api"
Nov 01 21:16:02.260390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260390    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" containerName="maas-api"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260396    5066 cpu_manager.go:414] "containerMap: removing container" podUID="a0e40375-406f-4362-b9ff-c365a907d466" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260401    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e40375-406f-4362-b9ff-c365a907d466" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260408    5066 cpu_manager.go:414] "containerMap: removing container" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260413    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260418    5066 cpu_manager.go:414] "containerMap: removing container" podUID="d9299ecf-8d8d-4811-ba57-ff3725601217" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260422    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9299ecf-8d8d-4811-ba57-ff3725601217" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260432    5066 cpu_manager.go:414] "containerMap: removing container" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260437    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260450    5066 cpu_manager.go:414] "containerMap: removing container" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260454    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260460    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260464    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260520    5066 cpu_manager.go:414] "containerMap: removing container" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260527    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260618    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="caff215e-7392-40d2-b79e-9cb3227d92e2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260630    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4047a62-1230-47bb-9b80-7ed53390deb2" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260636    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cccca0e-4a10-4c39-a2c3-7af0ca2e7511" containerName="payload-pre-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260643    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="292b3de2-9c16-4c09-892c-2c72992231b4" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260652    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8f9686-1afa-4299-9583-d92a035a3c67" containerName="istio-proxy"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260658    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="55054cc1-2b4a-452e-84e7-ff27b6e9cc71" containerName="maas-api"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260663    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d4897f-7ea5-4cf2-a861-6a35f1979dfb" containerName="maas-api"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260668    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e40375-406f-4362-b9ff-c365a907d466" containerName="payload-processing"
Nov 01 21:16:02.260661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260675    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0f6eb6-5d80-4626-a40a-a175b4bf0a2c" containerName="payload-processing"
Nov 01 21:16:02.261701 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.260681    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9299ecf-8d8d-4811-ba57-ff3725601217" containerName="payload-processing"
Nov 01 21:16:02.265190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.265171    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.317420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.317353    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3e843fb4-openshift-default-dockercfg-gtm2n\""
Nov 01 21:16:02.370448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.370414    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:16:02.401134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401094    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401132    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401241    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401286    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h9l\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401532 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401414    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401532 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401476    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401531    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401562    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.401643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.401594    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.502763 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502725    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.502763 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502770    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502788    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502906    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502963    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.502985    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503054    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503087    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h9l\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503142    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503188    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503365    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503392    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503447    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.503865 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.503844    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.505295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.505265    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.505405 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.505348    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.559393 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.559322    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.559544 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.559424    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h9l\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\") pod \"e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") " pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:02.577567 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:02.577539    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:03.003440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.003412    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:16:03.005478 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:16:03.005454    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161dc9f9_0a43_43eb_ab79_4a40c292d1ee.slice/crio-25e89d5291eaab9e6af2a26952b412e10137472e968082d36e2c10d16cf3580d WatchSource:0}: Error finding container 25e89d5291eaab9e6af2a26952b412e10137472e968082d36e2c10d16cf3580d: Status 404 returned error can't find the container with id 25e89d5291eaab9e6af2a26952b412e10137472e968082d36e2c10d16cf3580d
Nov 01 21:16:03.007648 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.007617    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:03.007762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.007693    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:03.007762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.007739    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:03.372480 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.372385    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerStarted","Data":"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"}
Nov 01 21:16:03.372480 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.372456    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerStarted","Data":"25e89d5291eaab9e6af2a26952b412e10137472e968082d36e2c10d16cf3580d"}
Nov 01 21:16:03.487712 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.487649    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podStartSLOduration=1.487632525 podStartE2EDuration="1.487632525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:03.486880063 +0000 UTC m=+4416.871777652" watchObservedRunningTime="2026-11-01 21:16:03.487632525 +0000 UTC m=+4416.872530117"
Nov 01 21:16:03.577862 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:03.577828    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:04.583798 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:04.583762    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:05.384725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:05.384693    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:05.385757 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:05.385735    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:09.142856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.142821    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:09.146479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.146461    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.204954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.204902    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:09.249590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.249556    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3e843fb4-dockercfg-9s826\""
Nov 01 21:16:09.249590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.249558    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3e843fb4\""
Nov 01 21:16:09.267477 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.267445    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.267686 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.267518    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv27b\" (UniqueName: \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.296848 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.296812    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:09.300369 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.300348    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.355748 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.355714    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:09.368575 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.368540    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.369059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.368609    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv27b\" (UniqueName: \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.369252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.369231    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.425261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.425183    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv27b\" (UniqueName: \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\") pod \"payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.460026 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.459993    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:09.469508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.469467    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.469643 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.469587    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622kr\" (UniqueName: \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.570960 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.570919    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.571167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.570973    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622kr\" (UniqueName: \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.571741 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.571713    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.627049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.627013    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-622kr\" (UniqueName: \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\") pod \"payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") " pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:09.830764 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.830733    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:09.832606 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:16:09.832578    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c8e5af_8aad_4f31_87e0_4223fd77040b.slice/crio-69664edf0636608e632aa1ba5cb9314a6c4f7a803ec39821a094930377a6d646 WatchSource:0}: Error finding container 69664edf0636608e632aa1ba5cb9314a6c4f7a803ec39821a094930377a6d646: Status 404 returned error can't find the container with id 69664edf0636608e632aa1ba5cb9314a6c4f7a803ec39821a094930377a6d646
Nov 01 21:16:09.913369 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:09.913339    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:10.292061 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.284906    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:10.412031 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.411990    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" event={"ID":"ad1bf45d-d624-4abb-89fe-726f8c955fd5","Type":"ContainerStarted","Data":"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"}
Nov 01 21:16:10.412242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.412058    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:10.412242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.412083    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" event={"ID":"ad1bf45d-d624-4abb-89fe-726f8c955fd5","Type":"ContainerStarted","Data":"0d9e6ba63f49002518245be3c24f01857d1fc431f642d5110888fff11edca38e"}
Nov 01 21:16:10.413481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.413457    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" event={"ID":"20c8e5af-8aad-4f31-87e0-4223fd77040b","Type":"ContainerStarted","Data":"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"}
Nov 01 21:16:10.413630 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.413525    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" event={"ID":"20c8e5af-8aad-4f31-87e0-4223fd77040b","Type":"ContainerStarted","Data":"69664edf0636608e632aa1ba5cb9314a6c4f7a803ec39821a094930377a6d646"}
Nov 01 21:16:10.413630 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.413554    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:10.526487 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.526430    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" podStartSLOduration=1.5264166700000001 podStartE2EDuration="1.52641667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:10.523950629 +0000 UTC m=+4423.908848218" watchObservedRunningTime="2026-11-01 21:16:10.52641667 +0000 UTC m=+4423.911314260"
Nov 01 21:16:10.637410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:10.637255    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" podStartSLOduration=1.637196136 podStartE2EDuration="1.637196136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:10.636073931 +0000 UTC m=+4424.020971552" watchObservedRunningTime="2026-11-01 21:16:10.637196136 +0000 UTC m=+4424.022093731"
Nov 01 21:16:11.631949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.631914    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:16:11.635619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.635596    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.738299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.738262    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:16:11.738479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.738326    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3e843fb4-serving-cert\""
Nov 01 21:16:11.738479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.738343    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:16:11.738479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.738360    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:16:11.738479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.738396    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:16:11.740811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.740793    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:16:11.792633 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.792584    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.792803 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.792641    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.792849 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.792806    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.792889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.792850    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkpf\" (UniqueName: \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.893640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.893530    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkpf\" (UniqueName: \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.893640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.893611    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.893893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.893647    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.893893 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:11.893765    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-serving-cert: secret "maas-api-e2e-ait-3e843fb4-serving-cert" not found
Nov 01 21:16:11.893893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.893814    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.893893 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:11.893846    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls podName:789b6671-35ff-4f2a-b190-08ba17455ee8 nodeName:}" failed. No retries permitted until 2026-11-01 21:16:12.393824829 +0000 UTC m=+4425.778722403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls") pod "maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" (UID: "789b6671-35ff-4f2a-b190-08ba17455ee8") : secret "maas-api-e2e-ait-3e843fb4-serving-cert" not found
Nov 01 21:16:11.894339 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.894316    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.896274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.896250    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:11.950646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:11.950616    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkpf\" (UniqueName: \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:12.399570 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:12.399525    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:12.401993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:12.401963    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") pod \"maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:12.550981 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:12.550934    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:12.925342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:12.925317    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:16:12.927133 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:16:12.927106    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789b6671_35ff_4f2a_b190_08ba17455ee8.slice/crio-192ac025bf71dd7a56ac9ada5a536a9f68b2d95afa3a3464de9403d6a24e5228 WatchSource:0}: Error finding container 192ac025bf71dd7a56ac9ada5a536a9f68b2d95afa3a3464de9403d6a24e5228: Status 404 returned error can't find the container with id 192ac025bf71dd7a56ac9ada5a536a9f68b2d95afa3a3464de9403d6a24e5228
Nov 01 21:16:12.928527 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:12.928508    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:16:13.430525 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:13.430468    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" event={"ID":"789b6671-35ff-4f2a-b190-08ba17455ee8","Type":"ContainerStarted","Data":"192ac025bf71dd7a56ac9ada5a536a9f68b2d95afa3a3464de9403d6a24e5228"}
Nov 01 21:16:14.438145 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:14.438104    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" event={"ID":"789b6671-35ff-4f2a-b190-08ba17455ee8","Type":"ContainerStarted","Data":"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"}
Nov 01 21:16:14.438600 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:14.438176    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:14.555393 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:14.555327    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" podStartSLOduration=2.644509658 podStartE2EDuration="3.555306411s" totalImagesPullingTime="910.796753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:16:14.554095602 +0000 UTC m=+4427.938993191" watchObservedRunningTime="2026-11-01 21:16:14.555306411 +0000 UTC m=+4427.940204001"
Nov 01 21:16:20.449695 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:20.449663    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:21.422748 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:21.422719    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:21.422911 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:21.422803    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:21.629058 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:21.629008    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:16:28.042830 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:28.042797    5066 scope.go:122] "RemoveContainer" containerID="efcbc8a91630a1ab120ed7909f7231732cc561868610021e5237f6c48c3c9df7"
Nov 01 21:16:29.543858 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.543825    5066 generic.go:356] "Generic (PLEG): container finished" podID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerID="74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869" exitCode=0
Nov 01 21:16:29.544315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.543896    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerDied","Data":"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"}
Nov 01 21:16:29.544315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.544264    5066 scope.go:122] "RemoveContainer" containerID="74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"
Nov 01 21:16:29.545389 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.545360    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:29.545506 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.545447    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:29.545639 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:29.545517    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:16:30.551158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:30.551125    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerStarted","Data":"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"}
Nov 01 21:16:30.585519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:30.582617    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:30.585519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:30.582750    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 21:16:30.585519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:30.582794    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 21:16:31.581455 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:31.580967    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 21:16:31.581455 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:31.581039    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 21:16:32.581966 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:32.581922    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 21:16:32.582621 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:32.582586    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 21:16:33.584518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:33.583626    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 21:16:33.584518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:33.583694    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 21:16:34.582518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:34.580895    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 21:16:34.582518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:34.580963    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 21:16:35.384953 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:35.384908    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:35.586778 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:35.586749    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:35.596874 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:35.596848    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:16:51.627793 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:51.627756    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:16:53.265481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:53.265438    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:53.369879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:53.369841    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:58.266310 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.266256    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" podUID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" containerName="payload-processing" containerID="cri-o://023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e" gracePeriod=25
Nov 01 21:16:58.370204 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.370132    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" podUID="20c8e5af-8aad-4f31-87e0-4223fd77040b" containerName="payload-pre-processing" containerID="cri-o://023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5" gracePeriod=25
Nov 01 21:16:58.521131 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.521057    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:58.627702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.627675    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:58.669477 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.669444    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\") pod \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") "
Nov 01 21:16:58.669668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.669486    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\" (UniqueName: \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\") pod \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\" (UID: \"ad1bf45d-d624-4abb-89fe-726f8c955fd5\") "
Nov 01 21:16:58.669954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.669932    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume" pod "ad1bf45d-d624-4abb-89fe-726f8c955fd5" (UID: "ad1bf45d-d624-4abb-89fe-726f8c955fd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:58.671699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.671670    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr" pod "ad1bf45d-d624-4abb-89fe-726f8c955fd5" (UID: "ad1bf45d-d624-4abb-89fe-726f8c955fd5"). InnerVolumeSpecName "kube-api-access-622kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:58.729791 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.729754    5066 generic.go:356] "Generic (PLEG): container finished" podID="20c8e5af-8aad-4f31-87e0-4223fd77040b" containerID="023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5" exitCode=0
Nov 01 21:16:58.729975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.729858    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"
Nov 01 21:16:58.731271 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.731244    5066 generic.go:356] "Generic (PLEG): container finished" podID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" containerID="023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e" exitCode=0
Nov 01 21:16:58.731417 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.731297    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"
Nov 01 21:16:58.732944 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.732922    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" event={"ID":"20c8e5af-8aad-4f31-87e0-4223fd77040b","Type":"ContainerDied","Data":"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"}
Nov 01 21:16:58.733055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.732968    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp" event={"ID":"20c8e5af-8aad-4f31-87e0-4223fd77040b","Type":"ContainerDied","Data":"69664edf0636608e632aa1ba5cb9314a6c4f7a803ec39821a094930377a6d646"}
Nov 01 21:16:58.733055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.732987    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" event={"ID":"ad1bf45d-d624-4abb-89fe-726f8c955fd5","Type":"ContainerDied","Data":"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"}
Nov 01 21:16:58.733055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.733004    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds" event={"ID":"ad1bf45d-d624-4abb-89fe-726f8c955fd5","Type":"ContainerDied","Data":"0d9e6ba63f49002518245be3c24f01857d1fc431f642d5110888fff11edca38e"}
Nov 01 21:16:58.733055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.733023    5066 scope.go:122] "RemoveContainer" containerID="023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"
Nov 01 21:16:58.743401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.743384    5066 scope.go:122] "RemoveContainer" containerID="023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"
Nov 01 21:16:58.744075 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:58.743670    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5\": container with ID starting with 023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5 not found: ID does not exist" containerID="023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"
Nov 01 21:16:58.744244 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.744216    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5"} err="failed to get container status \"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5\": rpc error: code = NotFound desc = could not find container \"023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5\": container with ID starting with 023029405c80bb67d178a32dde6f341e4dbc3d541149011bf751cac4603cc3b5 not found: ID does not exist"
Nov 01 21:16:58.744341 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.744329    5066 scope.go:122] "RemoveContainer" containerID="023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"
Nov 01 21:16:58.765596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.765569    5066 scope.go:122] "RemoveContainer" containerID="023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"
Nov 01 21:16:58.765951 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:58.765925    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e\": container with ID starting with 023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e not found: ID does not exist" containerID="023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"
Nov 01 21:16:58.766033 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.765960    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e"} err="failed to get container status \"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e\": rpc error: code = NotFound desc = could not find container \"023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e\": container with ID starting with 023e49cedabcfb0ac0b60b90c88138618b1d82693480765745465626e173917e not found: ID does not exist"
Nov 01 21:16:58.770323 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770296    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\") pod \"20c8e5af-8aad-4f31-87e0-4223fd77040b\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") "
Nov 01 21:16:58.770407 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770349    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\" (UniqueName: \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\") pod \"20c8e5af-8aad-4f31-87e0-4223fd77040b\" (UID: \"20c8e5af-8aad-4f31-87e0-4223fd77040b\") "
Nov 01 21:16:58.770759 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770731    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume" pod "20c8e5af-8aad-4f31-87e0-4223fd77040b" (UID: "20c8e5af-8aad-4f31-87e0-4223fd77040b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:58.770880 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770862    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1bf45d-d624-4abb-89fe-726f8c955fd5-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:58.770934 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770888    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-622kr\" (UniqueName: \"kubernetes.io/projected/ad1bf45d-d624-4abb-89fe-726f8c955fd5-kube-api-access-622kr\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:58.770934 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.770902    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c8e5af-8aad-4f31-87e0-4223fd77040b-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:58.772518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.772456    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b" pod "20c8e5af-8aad-4f31-87e0-4223fd77040b" (UID: "20c8e5af-8aad-4f31-87e0-4223fd77040b"). InnerVolumeSpecName "kube-api-access-cv27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:58.872074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.872035    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv27b\" (UniqueName: \"kubernetes.io/projected/20c8e5af-8aad-4f31-87e0-4223fd77040b-kube-api-access-cv27b\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:58.915277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.915233    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:58.919996 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:58.919956    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e843fb4-5c8d5f7b9-crwds"]
Nov 01 21:16:59.085710 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.085678    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:16:59.085912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.085893    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" podUID="789b6671-35ff-4f2a-b190-08ba17455ee8" containerName="maas-api" containerID="cri-o://a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e" gracePeriod=30
Nov 01 21:16:59.149120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.149084    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:59.153237 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.153203    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e843fb4-576b556d59-8kmpp"]
Nov 01 21:16:59.339286 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.339264    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:59.477201 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477166    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") pod \"789b6671-35ff-4f2a-b190-08ba17455ee8\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") "
Nov 01 21:16:59.477380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477247    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\") pod \"789b6671-35ff-4f2a-b190-08ba17455ee8\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") "
Nov 01 21:16:59.477380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477291    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\" (UniqueName: \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\") pod \"789b6671-35ff-4f2a-b190-08ba17455ee8\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") "
Nov 01 21:16:59.477540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477390    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\") pod \"789b6671-35ff-4f2a-b190-08ba17455ee8\" (UID: \"789b6671-35ff-4f2a-b190-08ba17455ee8\") "
Nov 01 21:16:59.477652 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477621    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca" pod "789b6671-35ff-4f2a-b190-08ba17455ee8" (UID: "789b6671-35ff-4f2a-b190-08ba17455ee8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:16:59.477776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.477751    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/789b6671-35ff-4f2a-b190-08ba17455ee8-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:59.480062 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.480030    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls" pod "789b6671-35ff-4f2a-b190-08ba17455ee8" (UID: "789b6671-35ff-4f2a-b190-08ba17455ee8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:59.480062 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.480045    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs" pod "789b6671-35ff-4f2a-b190-08ba17455ee8" (UID: "789b6671-35ff-4f2a-b190-08ba17455ee8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:16:59.480190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.480082    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf" pod "789b6671-35ff-4f2a-b190-08ba17455ee8" (UID: "789b6671-35ff-4f2a-b190-08ba17455ee8"). InnerVolumeSpecName "kube-api-access-rxkpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:16:59.578589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.578482    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkpf\" (UniqueName: \"kubernetes.io/projected/789b6671-35ff-4f2a-b190-08ba17455ee8-kube-api-access-rxkpf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:59.578589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.578543    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:59.578589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.578557    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/789b6671-35ff-4f2a-b190-08ba17455ee8-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:16:59.737971 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.737936    5066 generic.go:356] "Generic (PLEG): container finished" podID="789b6671-35ff-4f2a-b190-08ba17455ee8" containerID="a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e" exitCode=0
Nov 01 21:16:59.738154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.737979    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"
Nov 01 21:16:59.738154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.738023    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" event={"ID":"789b6671-35ff-4f2a-b190-08ba17455ee8","Type":"ContainerDied","Data":"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"}
Nov 01 21:16:59.738154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.738067    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7" event={"ID":"789b6671-35ff-4f2a-b190-08ba17455ee8","Type":"ContainerDied","Data":"192ac025bf71dd7a56ac9ada5a536a9f68b2d95afa3a3464de9403d6a24e5228"}
Nov 01 21:16:59.738154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.738091    5066 scope.go:122] "RemoveContainer" containerID="a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"
Nov 01 21:16:59.747908 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.747891    5066 scope.go:122] "RemoveContainer" containerID="a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"
Nov 01 21:16:59.748194 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:16:59.748174    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e\": container with ID starting with a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e not found: ID does not exist" containerID="a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"
Nov 01 21:16:59.748253 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.748204    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e"} err="failed to get container status \"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e\": rpc error: code = NotFound desc = could not find container \"a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e\": container with ID starting with a288eb9e2878e5745b1819ba8bb8cb4c14c55b97119e5f4c738658aa4f516b1e not found: ID does not exist"
Nov 01 21:16:59.901928 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.901840    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:16:59.903757 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:16:59.903736    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3e843fb4-7847dfd648-x6zc7"]
Nov 01 21:17:00.730080 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:00.730034    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c8e5af-8aad-4f31-87e0-4223fd77040b" path="/var/lib/kubelet/pods/20c8e5af-8aad-4f31-87e0-4223fd77040b/volumes"
Nov 01 21:17:00.730651 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:00.730628    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789b6671-35ff-4f2a-b190-08ba17455ee8" path="/var/lib/kubelet/pods/789b6671-35ff-4f2a-b190-08ba17455ee8/volumes"
Nov 01 21:17:00.731015 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:00.730998    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" path="/var/lib/kubelet/pods/ad1bf45d-d624-4abb-89fe-726f8c955fd5/volumes"
Nov 01 21:17:02.799615 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:02.799582    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:17:02.800042 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:02.799788    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" containerID="cri-o://53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909" gracePeriod=30
Nov 01 21:17:05.598650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:05.598610    5066 patch_prober.go:29] interesting pod/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:17:05.599135 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:05.598689    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:17:08.051181 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.051156    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:17:08.052278 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052257    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052330    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052363    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052406    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052440    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052477    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052674 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052533    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052674 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052582    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052674 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052610    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\") pod \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\" (UID: \"161dc9f9-0a43-43eb-ab79-4a40c292d1ee\") "
Nov 01 21:17:08.052810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052667    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:08.052810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052727    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:17:08.052917 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052834    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:08.052975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.052952    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:08.053033 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053012    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:08.053106 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053091    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.053164 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053112    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.053164 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053128    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.053164 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053143    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.053164 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.053158    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.054853 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.054832    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:17:08.054941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.054918    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "kube-api-access-f9h9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:08.055130 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.055115    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:17:08.055263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.055240    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token" pod "161dc9f9-0a43-43eb-ab79-4a40c292d1ee" (UID: "161dc9f9-0a43-43eb-ab79-4a40c292d1ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:17:08.153841 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.153774    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.153841 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.153800    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h9l\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-kube-api-access-f9h9l\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.153841 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.153811    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.153841 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.153819    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161dc9f9-0a43-43eb-ab79-4a40c292d1ee-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:17:08.793435 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.793399    5066 generic.go:356] "Generic (PLEG): container finished" podID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerID="53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909" exitCode=0
Nov 01 21:17:08.793609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.793449    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"
Nov 01 21:17:08.793609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.793479    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerDied","Data":"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"}
Nov 01 21:17:08.793609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.793546    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9" event={"ID":"161dc9f9-0a43-43eb-ab79-4a40c292d1ee","Type":"ContainerDied","Data":"25e89d5291eaab9e6af2a26952b412e10137472e968082d36e2c10d16cf3580d"}
Nov 01 21:17:08.793609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.793573    5066 scope.go:122] "RemoveContainer" containerID="53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"
Nov 01 21:17:08.802522 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.802503    5066 scope.go:122] "RemoveContainer" containerID="74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"
Nov 01 21:17:08.810158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.810144    5066 scope.go:122] "RemoveContainer" containerID="53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"
Nov 01 21:17:08.810428 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:17:08.810408    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909\": container with ID starting with 53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909 not found: ID does not exist" containerID="53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"
Nov 01 21:17:08.810520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.810434    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909"} err="failed to get container status \"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909\": rpc error: code = NotFound desc = could not find container \"53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909\": container with ID starting with 53df9f358a538f1f9b6e9280e5eea2624f93434025645a4de6bfb34df40d8909 not found: ID does not exist"
Nov 01 21:17:08.810520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.810449    5066 scope.go:122] "RemoveContainer" containerID="74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"
Nov 01 21:17:08.810683 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:17:08.810666    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869\": container with ID starting with 74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869 not found: ID does not exist" containerID="74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"
Nov 01 21:17:08.810722 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.810688    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869"} err="failed to get container status \"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869\": rpc error: code = NotFound desc = could not find container \"74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869\": container with ID starting with 74d56d1b4eb25aa9050b5303fff271df37119253ce86fd39097b83c7edcd3869 not found: ID does not exist"
Nov 01 21:17:08.955877 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:08.955844    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:17:09.005760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:09.005670    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3e843fb4-openshift-default-bdc5857d7-v4gt9"]
Nov 01 21:17:10.729780 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:10.729746    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" path="/var/lib/kubelet/pods/161dc9f9-0a43-43eb-ab79-4a40c292d1ee/volumes"
Nov 01 21:17:21.627640 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:17:21.627600    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:17:27.441098 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:27.441071    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:17:27.500898 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:27.500872    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:17:36.102073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102037    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102425    5066 cpu_manager.go:414] "containerMap: removing container" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102435    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102458    5066 cpu_manager.go:414] "containerMap: removing container" podUID="20c8e5af-8aad-4f31-87e0-4223fd77040b" containerName="payload-pre-processing"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102463    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c8e5af-8aad-4f31-87e0-4223fd77040b" containerName="payload-pre-processing"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102475    5066 cpu_manager.go:414] "containerMap: removing container" podUID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" containerName="payload-processing"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102480    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" containerName="payload-processing"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102504    5066 cpu_manager.go:414] "containerMap: removing container" podUID="789b6671-35ff-4f2a-b190-08ba17455ee8" containerName="maas-api"
Nov 01 21:17:36.102531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102512    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b6671-35ff-4f2a-b190-08ba17455ee8" containerName="maas-api"
Nov 01 21:17:36.102844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102589    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1bf45d-d624-4abb-89fe-726f8c955fd5" containerName="payload-processing"
Nov 01 21:17:36.102844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102599    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b6671-35ff-4f2a-b190-08ba17455ee8" containerName="maas-api"
Nov 01 21:17:36.102844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102604    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.102844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102612    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.102844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.102621    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c8e5af-8aad-4f31-87e0-4223fd77040b" containerName="payload-pre-processing"
Nov 01 21:17:36.107398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.107382    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180200 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180173    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180215    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180240    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180422 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180351    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180422 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180383    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180454    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180485    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2hh\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180522    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.180673 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.180651    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.208884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.208848    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 21:17:36.211060 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.211039    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:17:36.216435 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.216413    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:17:36.216830 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.216817    5066 cpu_manager.go:414] "containerMap: removing container" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.216879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.216830    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161dc9f9-0a43-43eb-ab79-4a40c292d1ee" containerName="istio-proxy"
Nov 01 21:17:36.219984 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.219952    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282060 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282025    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282224 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282075    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282224 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282175    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282224 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282216    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282257    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282292    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282338    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282376    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282420    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282457    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282516    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2hh\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282517    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282664    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282734    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282774    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282796    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bq4\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282834    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282877    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.282945 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.282913    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.283751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.283351    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.283751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.283377    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.283751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.283385    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.283751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.283701    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.285002 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.284979    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.285236 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.285218    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.351846 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.351808    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.352783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.352724    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2hh\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pgrt7\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.383502 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383461    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383630 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383592    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383642    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383677    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383714    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383736    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bq4\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383770    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.383820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383808    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384005 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383836    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384005 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.383894    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.384169    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384462 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.384438    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384588 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.384516    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.384588 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.384563    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.386571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.386552    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.386773 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.386754    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.420698 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.420673    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:36.440614 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.440590    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.440712 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.440654    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bq4\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vdzv7\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.533158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.533127    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:36.565014 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.564985    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:17:36.844319 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.844293    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:17:36.846035 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:17:36.846010    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fab25f_0b64_4191_8278_79316172a3b0.slice/crio-dd033650695d839c7c648966912498e94ee491d26c7576956c1d0cb16675ea49 WatchSource:0}: Error finding container dd033650695d839c7c648966912498e94ee491d26c7576956c1d0cb16675ea49: Status 404 returned error can't find the container with id dd033650695d839c7c648966912498e94ee491d26c7576956c1d0cb16675ea49
Nov 01 21:17:36.848065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.848032    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:36.848144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.848092    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:36.848144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.848120    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:36.946257 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.946221    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" event={"ID":"69fab25f-0b64-4191-8278-79316172a3b0","Type":"ContainerStarted","Data":"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"}
Nov 01 21:17:36.946402 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:36.946284    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" event={"ID":"69fab25f-0b64-4191-8278-79316172a3b0","Type":"ContainerStarted","Data":"dd033650695d839c7c648966912498e94ee491d26c7576956c1d0cb16675ea49"}
Nov 01 21:17:37.061276 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.061154    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podStartSLOduration=1.061140299 podStartE2EDuration="1.061140299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:37.060026252 +0000 UTC m=+4510.444923843" watchObservedRunningTime="2026-11-01 21:17:37.061140299 +0000 UTC m=+4510.446037889"
Nov 01 21:17:37.107992 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.107961    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:17:37.111346 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:17:37.111310    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52ff78c_5fad_4952_93b4_c5a6d031159e.slice/crio-6b330f016071e314c0ca39724b3021c3e3ba312e293e3bfd4406cfdb11260b1d WatchSource:0}: Error finding container 6b330f016071e314c0ca39724b3021c3e3ba312e293e3bfd4406cfdb11260b1d: Status 404 returned error can't find the container with id 6b330f016071e314c0ca39724b3021c3e3ba312e293e3bfd4406cfdb11260b1d
Nov 01 21:17:37.113594 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.113563    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:37.113728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.113628    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:37.113728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.113659    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:17:37.421854 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.421772    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:37.423330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.423302    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:37.423475 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.423350    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:37.954648 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.954599    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" event={"ID":"e52ff78c-5fad-4952-93b4-c5a6d031159e","Type":"ContainerStarted","Data":"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"}
Nov 01 21:17:37.954866 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:37.954670    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" event={"ID":"e52ff78c-5fad-4952-93b4-c5a6d031159e","Type":"ContainerStarted","Data":"6b330f016071e314c0ca39724b3021c3e3ba312e293e3bfd4406cfdb11260b1d"}
Nov 01 21:17:38.421017 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:38.420986    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:38.421455 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:38.421037    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:38.533503 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:38.533452    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:38.535099 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:38.535057    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:38.535211 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:38.535121    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:39.423950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:39.423907    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:39.424355 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:39.423983    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:39.534073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:39.534035    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:39.534249 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:39.534091    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:40.421473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:40.421437    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:40.421675 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:40.421518    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:40.533471 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:40.533433    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:40.533874 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:40.533525    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:41.421028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:41.420990    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:41.421209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:41.421043    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:41.534175 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:41.534127    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:41.534585 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:41.534191    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:42.421601 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:42.421564    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:42.421790 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:42.421618    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:42.533665 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:42.533616    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:42.533834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:42.533713    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:43.421349 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:43.421309    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:43.421767 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:43.421381    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:43.534044 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:43.534003    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:43.534231 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:43.534068    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:44.421441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:44.421398    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 01 21:17:44.421886 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:44.421471    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 01 21:17:44.534037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:44.533995    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 01 21:17:44.534307 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:44.534062    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 01 21:17:45.993466 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:45.993430    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:45.994170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:45.994111    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:45.995853 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:45.995829    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:17:46.163423 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:46.163348    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podStartSLOduration=10.163328047 podStartE2EDuration="10.163328047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:17:38.086890541 +0000 UTC m=+4511.471788132" watchObservedRunningTime="2026-11-01 21:17:46.163328047 +0000 UTC m=+4519.548225639"
Nov 01 21:17:46.212270 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:46.212217    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:46.212680 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:46.212657    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:46.213376 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:17:46.213356    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:17:51.627649 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:17:51.627613    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:18:19.139303 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.139197    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"]
Nov 01 21:18:19.145011 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.144981    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.195453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.195408    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"]
Nov 01 21:18:19.198588 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.198544    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:18:19.198753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.198600    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:18:19.198753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.198631    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:18:19.198753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.198714    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\""
Nov 01 21:18:19.243147 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.243104    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.243330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.243242    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2nc\" (UniqueName: \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.243330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.243283    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.344241 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.344201    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.344426 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.344287    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2nc\" (UniqueName: \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.344426 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.344314    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.345094 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.345077    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.346728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.346705    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.400750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.400674    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2nc\" (UniqueName: \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\") pod \"maas-api-revoke-sub-f904bff9-4pwm9\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.459890 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.459854    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:19.835485 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:19.835459    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"]
Nov 01 21:18:19.837727 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:18:19.837698    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ada81_0d73_4c30_9c93_4edbcfed78f0.slice/crio-02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52 WatchSource:0}: Error finding container 02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52: Status 404 returned error can't find the container with id 02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52
Nov 01 21:18:20.224824 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:20.224787    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9" event={"ID":"9a6ada81-0d73-4c30-9c93-4edbcfed78f0","Type":"ContainerStarted","Data":"ba9afdebd9fe01a62cb889dedabed2c763264b78d7b7323e2366f3abab91a5bf"}
Nov 01 21:18:20.225315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:20.224845    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9" event={"ID":"9a6ada81-0d73-4c30-9c93-4edbcfed78f0","Type":"ContainerStarted","Data":"02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52"}
Nov 01 21:18:20.338577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:20.338525    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9" podStartSLOduration=1.338510075 podStartE2EDuration="1.338510075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:20.336162319 +0000 UTC m=+4553.721059908" watchObservedRunningTime="2026-11-01 21:18:20.338510075 +0000 UTC m=+4553.723407661"
Nov 01 21:18:21.232167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:21.232133    5066 generic.go:356] "Generic (PLEG): container finished" podID="9a6ada81-0d73-4c30-9c93-4edbcfed78f0" containerID="ba9afdebd9fe01a62cb889dedabed2c763264b78d7b7323e2366f3abab91a5bf" exitCode=0
Nov 01 21:18:21.232596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:21.232210    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9" event={"ID":"9a6ada81-0d73-4c30-9c93-4edbcfed78f0","Type":"ContainerDied","Data":"ba9afdebd9fe01a62cb889dedabed2c763264b78d7b7323e2366f3abab91a5bf"}
Nov 01 21:18:21.628734 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:18:21.628649    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:18:22.378339 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.378314    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:22.474257 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.474222    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\") pod \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") "
Nov 01 21:18:22.474439 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.474290    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\" (UniqueName: \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\") pod \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") "
Nov 01 21:18:22.474439 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.474321    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\") pod \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\" (UID: \"9a6ada81-0d73-4c30-9c93-4edbcfed78f0\") "
Nov 01 21:18:22.474746 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.474718    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca" pod "9a6ada81-0d73-4c30-9c93-4edbcfed78f0" (UID: "9a6ada81-0d73-4c30-9c93-4edbcfed78f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:18:22.476431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.476405    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca" pod "9a6ada81-0d73-4c30-9c93-4edbcfed78f0" (UID: "9a6ada81-0d73-4c30-9c93-4edbcfed78f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:18:22.476610 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.476570    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc" pod "9a6ada81-0d73-4c30-9c93-4edbcfed78f0" (UID: "9a6ada81-0d73-4c30-9c93-4edbcfed78f0"). InnerVolumeSpecName "kube-api-access-8j2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:18:22.575158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.575074    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-xks-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:18:22.575158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.575103    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j2nc\" (UniqueName: \"kubernetes.io/projected/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-kube-api-access-8j2nc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:18:22.575158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:22.575113    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6ada81-0d73-4c30-9c93-4edbcfed78f0-maas-api-openshift-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:18:23.244059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:23.244026    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"
Nov 01 21:18:23.244059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:23.244041    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9" event={"ID":"9a6ada81-0d73-4c30-9c93-4edbcfed78f0","Type":"ContainerDied","Data":"02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52"}
Nov 01 21:18:23.244278 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:23.244087    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02483163f081fc1906b7d46b96c3235870246e3d3eb25e1144540aa7645b8d52"
Nov 01 21:18:36.577406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.577365    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:18:36.577913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.577855    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9a6ada81-0d73-4c30-9c93-4edbcfed78f0" containerName="revoke-keys"
Nov 01 21:18:36.577913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.577867    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6ada81-0d73-4c30-9c93-4edbcfed78f0" containerName="revoke-keys"
Nov 01 21:18:36.577987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.577938    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6ada81-0d73-4c30-9c93-4edbcfed78f0" containerName="revoke-keys"
Nov 01 21:18:36.581350 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.581327    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.627093 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.627059    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:18:36.630635 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.630617    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.725681 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725649    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.725881 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725691    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.725881 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725712    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.725881 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725842    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725890    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.725981    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726020    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pt2j\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726061    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726176    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726210    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726262    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726333 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726290    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726333 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726310    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726444 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726336    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726444 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726405    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.726444 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726434    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726656 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726591    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.726656 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.726637    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvtr\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.767750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.767715    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:18:36.827912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827821    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.827912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827867    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pt2j\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.827912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827893    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827933    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827959    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.827996    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828020    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828044    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828346    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828400    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828479 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828464    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828530    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828582    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828623    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvtr\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828655    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828700    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828709    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828723    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828752    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828791    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.828895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828811    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.829202 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.828976    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.829202 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829108    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.829377 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829355    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.829462 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829357    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.829582 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829561    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.829911 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829886    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.830009 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.829893    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.830912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.830889    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.831113 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.831088    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.831335 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.831314    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.831381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.831317    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.836504 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.836464    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:18:36.885987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.885593    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.885987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.885706    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pt2j\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.885987 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.885846    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvtr\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ljwmf\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:36.886296 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.886256    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fmf7p\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.895347 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.895274    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:36.943583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:36.943541    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:37.329196 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.329132    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:18:37.333048 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:18:37.333018    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33adf5c9_ce85_4cea_abdd_ae9834ebd472.slice/crio-df1458550e15771a35fbeae5c857d8f0882b49e6216c00aa36a21524a1e5fe70 WatchSource:0}: Error finding container df1458550e15771a35fbeae5c857d8f0882b49e6216c00aa36a21524a1e5fe70: Status 404 returned error can't find the container with id df1458550e15771a35fbeae5c857d8f0882b49e6216c00aa36a21524a1e5fe70
Nov 01 21:18:37.335747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.335715    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:37.335850 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.335775    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:37.335850 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.335805    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:37.380434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.380412    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:18:37.383030 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:18:37.382999    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc0490c_6130_47ff_ad1d_36a80b14741b.slice/crio-f9c18577d188c7f78365215a170eda976ab124ed888d369c56bc6c4e380962f4 WatchSource:0}: Error finding container f9c18577d188c7f78365215a170eda976ab124ed888d369c56bc6c4e380962f4: Status 404 returned error can't find the container with id f9c18577d188c7f78365215a170eda976ab124ed888d369c56bc6c4e380962f4
Nov 01 21:18:37.386094 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.386059    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:37.386191 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.386136    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:37.386191 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:37.386176    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:18:38.334021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.333864    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" event={"ID":"bfc0490c-6130-47ff-ad1d-36a80b14741b","Type":"ContainerStarted","Data":"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"}
Nov 01 21:18:38.334021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.333933    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" event={"ID":"bfc0490c-6130-47ff-ad1d-36a80b14741b","Type":"ContainerStarted","Data":"f9c18577d188c7f78365215a170eda976ab124ed888d369c56bc6c4e380962f4"}
Nov 01 21:18:38.335677 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.335644    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" event={"ID":"33adf5c9-ce85-4cea-abdd-ae9834ebd472","Type":"ContainerStarted","Data":"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"}
Nov 01 21:18:38.335811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.335707    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" event={"ID":"33adf5c9-ce85-4cea-abdd-ae9834ebd472","Type":"ContainerStarted","Data":"df1458550e15771a35fbeae5c857d8f0882b49e6216c00aa36a21524a1e5fe70"}
Nov 01 21:18:38.557563 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.557457    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podStartSLOduration=2.5574357450000003 podStartE2EDuration="2.557435745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:38.507116685 +0000 UTC m=+4571.892014277" watchObservedRunningTime="2026-11-01 21:18:38.557435745 +0000 UTC m=+4571.942333337"
Nov 01 21:18:38.720583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.705045    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podStartSLOduration=2.70501812 podStartE2EDuration="2.70501812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:38.688063523 +0000 UTC m=+4572.072961116" watchObservedRunningTime="2026-11-01 21:18:38.70501812 +0000 UTC m=+4572.089915709"
Nov 01 21:18:38.899519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.898474    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:38.905515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.901624    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:38.905515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.901688    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:38.945229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.945194    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:38.945654 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.945626    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:38.946678 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:38.946645    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:39.897955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:39.897912    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:39.898583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:39.898549    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:39.946024 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:39.945980    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:39.946319 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:39.946270    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:40.898519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:40.897957    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:40.898519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:40.898051    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:40.953520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:40.947526    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:40.953520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:40.947606    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:41.896519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:41.895912    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:41.896519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:41.895989    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:41.944929 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:41.944882    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:41.945340 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:41.944939    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:42.909645 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:42.909604    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:42.909950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:42.909919    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:42.944339 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:42.944304    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:42.944513 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:42.944361    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:43.896507 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:43.896457    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:43.896903 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:43.896553    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:43.944440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:43.944396    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:43.944626 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:43.944469    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:44.896219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:44.896174    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 21:18:44.896401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:44.896250    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 21:18:44.945016 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:44.944974    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 21:18:44.945407 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:44.945047    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 21:18:46.885855 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.885805    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:46.886466 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.886441    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:46.887205 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.887179    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:18:46.935898 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.935859    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fmf7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:18:46.936125 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.935924    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:18:46.979684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.979650    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:46.980441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.980410    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:46.981235 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:18:46.981211    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:18:51.778806 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:18:51.778754    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:19:21.627835 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:19:21.627798    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:19:43.411136 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.411035    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:19:43.415636 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.415611    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.470439 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.470397    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:19:43.528447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.517827    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-78927386-kserve-self-signed-certs\""
Nov 01 21:19:43.532868 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.532836    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.532977 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.532879    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.533219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.533187    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpmqf\" (UniqueName: \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.533291 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.533252    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.533381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.533365    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.533537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.533482    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634720 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634685    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634766    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634786    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634801    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634844    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpmqf\" (UniqueName: \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.634931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.634872    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.635826 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.635795    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.636007 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.635869    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.636153 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.636127    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.637487 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.637463    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.638525 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.638376    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.693315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.693216    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpmqf\" (UniqueName: \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\") pod \"e2e-prio-pool-78927386-kserve-6cb565f77f-996q7\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") " pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:43.733482 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:43.733445    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:44.142812 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:44.142783    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:19:44.144714 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:19:44.144685    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae72993_caac_4ba7_9bac_4bf48d440a20.slice/crio-683ea49b38a1e4f524673cf87c200863263c7576f1338dece3e9c5e18a49a767 WatchSource:0}: Error finding container 683ea49b38a1e4f524673cf87c200863263c7576f1338dece3e9c5e18a49a767: Status 404 returned error can't find the container with id 683ea49b38a1e4f524673cf87c200863263c7576f1338dece3e9c5e18a49a767
Nov 01 21:19:44.941135 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:44.941100    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" event={"ID":"5ae72993-caac-4ba7-9bac-4bf48d440a20","Type":"ContainerStarted","Data":"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"}
Nov 01 21:19:44.941542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:44.941150    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" event={"ID":"5ae72993-caac-4ba7-9bac-4bf48d440a20","Type":"ContainerStarted","Data":"683ea49b38a1e4f524673cf87c200863263c7576f1338dece3e9c5e18a49a767"}
Nov 01 21:19:45.054278 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:45.054218    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" podStartSLOduration=2.05420456 podStartE2EDuration="2.05420456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:19:45.052765611 +0000 UTC m=+4638.437663202" watchObservedRunningTime="2026-11-01 21:19:45.05420456 +0000 UTC m=+4638.439102149"
Nov 01 21:19:45.734417 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:45.734385    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:51.745984 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:19:51.745924    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:19:53.738744 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:53.738705    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:54.046473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:54.044825    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:19:54.309947 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:19:54.309848    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:20:21.627879 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:20:21.627838    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:20:48.635832 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.635797    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"]
Nov 01 21:20:48.639314 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.639298    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.697588 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.697554    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"]
Nov 01 21:20:48.741399 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.741367    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:20:48.741579 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.741417    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:20:48.741579 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.741443    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\""
Nov 01 21:20:48.741579 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.741521    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:20:48.772189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.772157    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljsl\" (UniqueName: \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.772189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.772192    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.772381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.772212    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.873664 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.873631    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljsl\" (UniqueName: \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.873664 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.873666    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.873900 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.873685    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.874300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.874278    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.876024 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.876001    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.930073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.930009    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljsl\" (UniqueName: \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\") pod \"maas-api-revoke-sub-d29ccddd-4bwgb\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:48.952238 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:48.952199    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:49.324779 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:49.324752    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"]
Nov 01 21:20:49.326639 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:20:49.326606    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7400cc17_91fc_4746_9c82_0a20e537c3d7.slice/crio-3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399 WatchSource:0}: Error finding container 3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399: Status 404 returned error can't find the container with id 3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399
Nov 01 21:20:49.379458 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:49.379432    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb" event={"ID":"7400cc17-91fc-4746-9c82-0a20e537c3d7","Type":"ContainerStarted","Data":"3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399"}
Nov 01 21:20:50.386000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:50.385913    5066 generic.go:356] "Generic (PLEG): container finished" podID="7400cc17-91fc-4746-9c82-0a20e537c3d7" containerID="21722fbd1fe9fb7b8666a804d7affad1294d1460183edfd40e501bffe7b93f88" exitCode=0
Nov 01 21:20:50.386000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:50.385988    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb" event={"ID":"7400cc17-91fc-4746-9c82-0a20e537c3d7","Type":"ContainerDied","Data":"21722fbd1fe9fb7b8666a804d7affad1294d1460183edfd40e501bffe7b93f88"}
Nov 01 21:20:51.535510 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.535469    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:51.599311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.599281    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\" (UniqueName: \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\") pod \"7400cc17-91fc-4746-9c82-0a20e537c3d7\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") "
Nov 01 21:20:51.599482 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.599376    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\") pod \"7400cc17-91fc-4746-9c82-0a20e537c3d7\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") "
Nov 01 21:20:51.599482 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.599450    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\") pod \"7400cc17-91fc-4746-9c82-0a20e537c3d7\" (UID: \"7400cc17-91fc-4746-9c82-0a20e537c3d7\") "
Nov 01 21:20:51.599834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.599800    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca" pod "7400cc17-91fc-4746-9c82-0a20e537c3d7" (UID: "7400cc17-91fc-4746-9c82-0a20e537c3d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:20:51.601442 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.601416    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl" pod "7400cc17-91fc-4746-9c82-0a20e537c3d7" (UID: "7400cc17-91fc-4746-9c82-0a20e537c3d7"). InnerVolumeSpecName "kube-api-access-bljsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:20:51.601619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.601598    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca" pod "7400cc17-91fc-4746-9c82-0a20e537c3d7" (UID: "7400cc17-91fc-4746-9c82-0a20e537c3d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:20:51.626958 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:20:51.626928    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:20:51.700691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.700613    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-xks-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:20:51.700691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.700640    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7400cc17-91fc-4746-9c82-0a20e537c3d7-maas-api-openshift-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:20:51.700691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:51.700652    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljsl\" (UniqueName: \"kubernetes.io/projected/7400cc17-91fc-4746-9c82-0a20e537c3d7-kube-api-access-bljsl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:20:52.399781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:52.399745    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb" event={"ID":"7400cc17-91fc-4746-9c82-0a20e537c3d7","Type":"ContainerDied","Data":"3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399"}
Nov 01 21:20:52.399781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:52.399784    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"
Nov 01 21:20:52.400001 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:20:52.399791    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a3d383dc35e5f3c1469e14c79851cc9fee5f96a4ee4db86ec9982bbc1e16399"
Nov 01 21:21:01.542535 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:01.542479    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:21:06.933822 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.933779    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"]
Nov 01 21:21:06.934334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.934198    5066 cpu_manager.go:414] "containerMap: removing container" podUID="7400cc17-91fc-4746-9c82-0a20e537c3d7" containerName="revoke-keys"
Nov 01 21:21:06.934334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.934209    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7400cc17-91fc-4746-9c82-0a20e537c3d7" containerName="revoke-keys"
Nov 01 21:21:06.934334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.934281    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="7400cc17-91fc-4746-9c82-0a20e537c3d7" containerName="revoke-keys"
Nov 01 21:21:06.937415 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.937390    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:06.988993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:06.988962    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"]
Nov 01 21:21:07.034607 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.034576    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:21:07.034755 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.034609    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:21:07.038408 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.038390    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\""
Nov 01 21:21:07.038460 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.038418    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:21:07.054711 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.054688    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.054816 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.054732    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.054943 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.054924    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkp74\" (UniqueName: \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.155547 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.155523    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp74\" (UniqueName: \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.155666 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.155568    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.155666 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.155618    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.156259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.156235    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.157773 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.157754    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.211182 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.211130    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp74\" (UniqueName: \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\") pod \"maas-api-revoke-sub-05308870-bp6jp\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.249963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.249942    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:07.624699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:07.624542    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"]
Nov 01 21:21:07.628799 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:21:07.628772    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295c7792_fd3a_4d24_8de1_16503c246a21.slice/crio-41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2 WatchSource:0}: Error finding container 41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2: Status 404 returned error can't find the container with id 41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2
Nov 01 21:21:08.499540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:08.499380    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp" event={"ID":"295c7792-fd3a-4d24-8de1-16503c246a21","Type":"ContainerStarted","Data":"1c01337619ce619ab368b692264d20b6f125a6a6c7eb7deab3a2996bee4c34d0"}
Nov 01 21:21:08.499540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:08.499445    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp" event={"ID":"295c7792-fd3a-4d24-8de1-16503c246a21","Type":"ContainerStarted","Data":"41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2"}
Nov 01 21:21:08.638518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:08.621311    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp" podStartSLOduration=2.621291965 podStartE2EDuration="2.621291965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:08.616874735 +0000 UTC m=+4722.001772327" watchObservedRunningTime="2026-11-01 21:21:08.621291965 +0000 UTC m=+4722.006189556"
Nov 01 21:21:09.542517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:09.537562    5066 generic.go:356] "Generic (PLEG): container finished" podID="295c7792-fd3a-4d24-8de1-16503c246a21" containerID="1c01337619ce619ab368b692264d20b6f125a6a6c7eb7deab3a2996bee4c34d0" exitCode=0
Nov 01 21:21:09.542517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:09.537694    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp" event={"ID":"295c7792-fd3a-4d24-8de1-16503c246a21","Type":"ContainerDied","Data":"1c01337619ce619ab368b692264d20b6f125a6a6c7eb7deab3a2996bee4c34d0"}
Nov 01 21:21:11.293533 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.292796    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:11.425256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.425228    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\") pod \"295c7792-fd3a-4d24-8de1-16503c246a21\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") "
Nov 01 21:21:11.425459 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.425273    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\" (UniqueName: \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\") pod \"295c7792-fd3a-4d24-8de1-16503c246a21\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") "
Nov 01 21:21:11.425459 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.425347    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\") pod \"295c7792-fd3a-4d24-8de1-16503c246a21\" (UID: \"295c7792-fd3a-4d24-8de1-16503c246a21\") "
Nov 01 21:21:11.425674 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.425650    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca" pod "295c7792-fd3a-4d24-8de1-16503c246a21" (UID: "295c7792-fd3a-4d24-8de1-16503c246a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:21:11.425820 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.425799    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-openshift-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:11.431027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.430865    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca" pod "295c7792-fd3a-4d24-8de1-16503c246a21" (UID: "295c7792-fd3a-4d24-8de1-16503c246a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:21:11.431027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.430980    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74" pod "295c7792-fd3a-4d24-8de1-16503c246a21" (UID: "295c7792-fd3a-4d24-8de1-16503c246a21"). InnerVolumeSpecName "kube-api-access-mkp74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:21:11.526796 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.526719    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkp74\" (UniqueName: \"kubernetes.io/projected/295c7792-fd3a-4d24-8de1-16503c246a21-kube-api-access-mkp74\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:11.526796 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.526748    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295c7792-fd3a-4d24-8de1-16503c246a21-maas-api-xks-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:11.554132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.554102    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp" event={"ID":"295c7792-fd3a-4d24-8de1-16503c246a21","Type":"ContainerDied","Data":"41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2"}
Nov 01 21:21:11.554247 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.554149    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41480ab7baf6f2cfb343edfce814d9090fa357991d022a133f84ac3978d765d2"
Nov 01 21:21:11.554247 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:11.554106    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"
Nov 01 21:21:16.566901 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.566838    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" podUID="5ae72993-caac-4ba7-9bac-4bf48d440a20" containerName="main" containerID="cri-o://c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc" gracePeriod=45
Nov 01 21:21:16.814620 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.814594    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:21:16.873119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873065    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\" (UniqueName: \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873116    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873254 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873143    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873254 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873165    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873254 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873193    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873385 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873339    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\") pod \"5ae72993-caac-4ba7-9bac-4bf48d440a20\" (UID: \"5ae72993-caac-4ba7-9bac-4bf48d440a20\") "
Nov 01 21:21:16.873581 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873454    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:21:16.873581 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873464    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:21:16.873735 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873593    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:21:16.873783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873737    5066 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-tmp-dir\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:16.873783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873752    5066 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-model-cache\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:16.873783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.873765    5066 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-home\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:16.875348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.875329    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:21:16.875629 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.875604    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:21:16.875745 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.875632    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf" pod "5ae72993-caac-4ba7-9bac-4bf48d440a20" (UID: "5ae72993-caac-4ba7-9bac-4bf48d440a20"). InnerVolumeSpecName "kube-api-access-wpmqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:21:16.975215 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.975185    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpmqf\" (UniqueName: \"kubernetes.io/projected/5ae72993-caac-4ba7-9bac-4bf48d440a20-kube-api-access-wpmqf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:16.975215 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.975210    5066 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae72993-caac-4ba7-9bac-4bf48d440a20-tls-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:16.975215 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:16.975220    5066 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae72993-caac-4ba7-9bac-4bf48d440a20-dshm\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:21:17.594611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.594576    5066 generic.go:356] "Generic (PLEG): container finished" podID="5ae72993-caac-4ba7-9bac-4bf48d440a20" containerID="c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc" exitCode=0
Nov 01 21:21:17.595027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.594619    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"
Nov 01 21:21:17.595027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.594625    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" event={"ID":"5ae72993-caac-4ba7-9bac-4bf48d440a20","Type":"ContainerDied","Data":"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"}
Nov 01 21:21:17.595027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.594662    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7" event={"ID":"5ae72993-caac-4ba7-9bac-4bf48d440a20","Type":"ContainerDied","Data":"683ea49b38a1e4f524673cf87c200863263c7576f1338dece3e9c5e18a49a767"}
Nov 01 21:21:17.595027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.594680    5066 scope.go:122] "RemoveContainer" containerID="c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"
Nov 01 21:21:17.604607 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.604590    5066 scope.go:122] "RemoveContainer" containerID="c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"
Nov 01 21:21:17.604959 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:21:17.604940    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc\": container with ID starting with c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc not found: ID does not exist" containerID="c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"
Nov 01 21:21:17.605024 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.604970    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc"} err="failed to get container status \"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc\": rpc error: code = NotFound desc = could not find container \"c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc\": container with ID starting with c9e6a1796f9dbac86af13f65b943690bb2f90525b42591cad45e1b1e150e9fbc not found: ID does not exist"
Nov 01 21:21:17.756537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.756506    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:21:17.806286 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:17.806256    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-78927386-kserve-6cb565f77f-996q7"]
Nov 01 21:21:18.735273 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:18.735223    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae72993-caac-4ba7-9bac-4bf48d440a20" path="/var/lib/kubelet/pods/5ae72993-caac-4ba7-9bac-4bf48d440a20/volumes"
Nov 01 21:21:19.757188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757155    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:21:19.757688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757614    5066 cpu_manager.go:414] "containerMap: removing container" podUID="5ae72993-caac-4ba7-9bac-4bf48d440a20" containerName="main"
Nov 01 21:21:19.757688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757627    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae72993-caac-4ba7-9bac-4bf48d440a20" containerName="main"
Nov 01 21:21:19.757688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757642    5066 cpu_manager.go:414] "containerMap: removing container" podUID="295c7792-fd3a-4d24-8de1-16503c246a21" containerName="revoke-keys"
Nov 01 21:21:19.757688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757648    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295c7792-fd3a-4d24-8de1-16503c246a21" containerName="revoke-keys"
Nov 01 21:21:19.757871 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757716    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae72993-caac-4ba7-9bac-4bf48d440a20" containerName="main"
Nov 01 21:21:19.757871 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.757726    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="295c7792-fd3a-4d24-8de1-16503c246a21" containerName="revoke-keys"
Nov 01 21:21:19.762012 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.761995    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.806870 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.806827    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.807175 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807153    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.807311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807293    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsw5\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.807485 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807466    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.807665 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807647    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.807857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807838    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.808001 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.807975    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.808134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.808119    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.808270 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.808253    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.817142 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.815881    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f3eeb7-openshift-default-dockercfg-nn4lb\""
Nov 01 21:21:19.866941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.866911    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.909053    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.915959    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916031    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsw5\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916308    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916402    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916548    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916616    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916655    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.916715    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.917087    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.917730    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.918068    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.918568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.918511    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.919696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.919663    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.920692 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.920670    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.920812 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.920787    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.968461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.968431    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:19.973548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:19.973528    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsw5\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\") pod \"e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") " pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:20.074749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.074683    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:20.507534 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.507510    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:21:20.509741 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:21:20.509716    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8fb376_4d54_4702_a66c_c227ac78a694.slice/crio-157f0f1c80ffc628f96f235777d1d689909ecbe909eec2a46701a04a7a799311 WatchSource:0}: Error finding container 157f0f1c80ffc628f96f235777d1d689909ecbe909eec2a46701a04a7a799311: Status 404 returned error can't find the container with id 157f0f1c80ffc628f96f235777d1d689909ecbe909eec2a46701a04a7a799311
Nov 01 21:21:20.511714 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.511681    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:20.511818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.511755    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:20.511818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.511797    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:20.618270 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.618231    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerStarted","Data":"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"}
Nov 01 21:21:20.618466 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.618296    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerStarted","Data":"157f0f1c80ffc628f96f235777d1d689909ecbe909eec2a46701a04a7a799311"}
Nov 01 21:21:20.734597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:20.734538    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podStartSLOduration=1.734521203 podStartE2EDuration="1.734521203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:20.732532394 +0000 UTC m=+4734.117429984" watchObservedRunningTime="2026-11-01 21:21:20.734521203 +0000 UTC m=+4734.119418794"
Nov 01 21:21:21.075155 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:21.075120    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:21.079936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:21.079910    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:21.624571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:21.624536    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:21.625769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:21.625607    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:21.627828 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:21:21.627799    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:21:27.197174 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.197143    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:21:27.202055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.202034    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.233865 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.233842    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4mr\" (UniqueName: \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.233964 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.233870    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.255211 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.255185    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:21:27.302016 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.301995    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f3eeb7\""
Nov 01 21:21:27.302016 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.302009    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-469613a7-dockercfg-qdthz\""
Nov 01 21:21:27.336515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.335816    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4mr\" (UniqueName: \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.336515 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.335878    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.336958 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.336927    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.354386 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.354346    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:21:27.357975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.357961    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.392702 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.392678    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4mr\" (UniqueName: \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.405522 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.405477    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:21:27.437285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.437256    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb24k\" (UniqueName: \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.437456 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.437437    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.515801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.515768    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:27.538344 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.538321    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.538470 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.538423    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb24k\" (UniqueName: \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.539160 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.539136    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.597951 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.597869    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb24k\" (UniqueName: \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.671021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.670986    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:27.889131 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:27.889103    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:21:27.891203 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:21:27.891176    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f9fcb8_befe_471c_b81d_5d029b9f6fda.slice/crio-159a589aad5ff00f5bb57830b97018c0757f1e52f2a841802e6d16d76cacc9aa WatchSource:0}: Error finding container 159a589aad5ff00f5bb57830b97018c0757f1e52f2a841802e6d16d76cacc9aa: Status 404 returned error can't find the container with id 159a589aad5ff00f5bb57830b97018c0757f1e52f2a841802e6d16d76cacc9aa
Nov 01 21:21:28.044284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.044246    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:21:28.045762 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:21:28.045740    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97124b5_3e4c_4051_b70f_7d86f60535ac.slice/crio-60f2776677ddef32c13ae7016cc271254e6f3bf902ce9c23c4a63cbafc2e3ad7 WatchSource:0}: Error finding container 60f2776677ddef32c13ae7016cc271254e6f3bf902ce9c23c4a63cbafc2e3ad7: Status 404 returned error can't find the container with id 60f2776677ddef32c13ae7016cc271254e6f3bf902ce9c23c4a63cbafc2e3ad7
Nov 01 21:21:28.696951 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.696832    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" event={"ID":"a97124b5-3e4c-4051-b70f-7d86f60535ac","Type":"ContainerStarted","Data":"e582574b8fc9b62587ea5fc730eee95d7b6a798286767d711ced4fe3074b550a"}
Nov 01 21:21:28.696951 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.696887    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:28.696951 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.696910    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" event={"ID":"a97124b5-3e4c-4051-b70f-7d86f60535ac","Type":"ContainerStarted","Data":"60f2776677ddef32c13ae7016cc271254e6f3bf902ce9c23c4a63cbafc2e3ad7"}
Nov 01 21:21:28.721028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.698456    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" event={"ID":"84f9fcb8-befe-471c-b81d-5d029b9f6fda","Type":"ContainerStarted","Data":"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"}
Nov 01 21:21:28.721028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.698528    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:28.721028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.698550    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" event={"ID":"84f9fcb8-befe-471c-b81d-5d029b9f6fda","Type":"ContainerStarted","Data":"159a589aad5ff00f5bb57830b97018c0757f1e52f2a841802e6d16d76cacc9aa"}
Nov 01 21:21:28.811282 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.811238    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" podStartSLOduration=1.811222025 podStartE2EDuration="1.811222025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:28.808759003 +0000 UTC m=+4742.193656593" watchObservedRunningTime="2026-11-01 21:21:28.811222025 +0000 UTC m=+4742.196119615"
Nov 01 21:21:28.925110 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:28.924736    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" podStartSLOduration=1.9247188130000001 podStartE2EDuration="1.924718813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:28.919846716 +0000 UTC m=+4742.304744306" watchObservedRunningTime="2026-11-01 21:21:28.924718813 +0000 UTC m=+4742.309616403"
Nov 01 21:21:29.778743 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.778709    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:21:29.782573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.782556    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.834917 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.834889    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f3eeb7-serving-cert\""
Nov 01 21:21:29.864729 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.864697    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.864910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.864891    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.864979 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.864932    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.865029 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.865013    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.885229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.884765    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:21:29.885229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.884888    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:21:29.885229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.884990    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:21:29.885229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.885136    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:21:29.887912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.887888    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:21:29.966570 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.966530    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.966722 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:21:29.966657    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-serving-cert: secret "maas-api-e2e-worker-single-main-f3eeb7-serving-cert" not found
Nov 01 21:21:29.966722 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:21:29.966720    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls podName:2a254c47-642e-4bdd-adc8-942185944eae nodeName:}" failed. No retries permitted until 2026-11-01 21:21:30.466703712 +0000 UTC m=+4743.851601308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls") pod "maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" (UID: "2a254c47-642e-4bdd-adc8-942185944eae") : secret "maas-api-e2e-worker-single-main-f3eeb7-serving-cert" not found
Nov 01 21:21:29.966815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.966737    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.966815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.966769    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.966815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.966808    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.967398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.967372    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:29.969180 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:29.969162    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:30.023406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:30.023383    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:30.472088 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:30.472048    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:30.474401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:30.474371    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:30.696879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:30.696844    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:31.073123 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:31.073099    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:21:31.075170 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:21:31.075135    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a254c47_642e_4bdd_adc8_942185944eae.slice/crio-5a8b5d9b398c60e3e50dba02f8c033e1e2a99b9d9bf94a0bb2b69090a407a4e1 WatchSource:0}: Error finding container 5a8b5d9b398c60e3e50dba02f8c033e1e2a99b9d9bf94a0bb2b69090a407a4e1: Status 404 returned error can't find the container with id 5a8b5d9b398c60e3e50dba02f8c033e1e2a99b9d9bf94a0bb2b69090a407a4e1
Nov 01 21:21:31.076464 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:31.076448    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:21:31.716797 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:31.716756    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" event={"ID":"2a254c47-642e-4bdd-adc8-942185944eae","Type":"ContainerStarted","Data":"5a8b5d9b398c60e3e50dba02f8c033e1e2a99b9d9bf94a0bb2b69090a407a4e1"}
Nov 01 21:21:32.729265 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:32.729231    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" event={"ID":"2a254c47-642e-4bdd-adc8-942185944eae","Type":"ContainerStarted","Data":"1617c8f08f136962f7b6c724bd045594331ff8c04bd7234b7669ee9296c7b371"}
Nov 01 21:21:32.729668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:32.729282    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:32.839464 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:32.839403    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" podStartSLOduration=2.933507992 podStartE2EDuration="3.839386001s" totalImagesPullingTime="905.878009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:21:32.838673395 +0000 UTC m=+4746.223570985" watchObservedRunningTime="2026-11-01 21:21:32.839386001 +0000 UTC m=+4746.224283591"
Nov 01 21:21:38.736677 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:38.736648    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:21:39.707621 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:39.707589    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:21:39.707793 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:39.707657    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:21:46.826254 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.826216    5066 generic.go:356] "Generic (PLEG): container finished" podID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerID="cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a" exitCode=0
Nov 01 21:21:46.826799 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.826291    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerDied","Data":"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"}
Nov 01 21:21:46.826799 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.826738    5066 scope.go:122] "RemoveContainer" containerID="cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"
Nov 01 21:21:46.827898 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.827855    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:46.827999 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.827931    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:46.827999 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:46.827970    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:21:47.833619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:47.833584    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerStarted","Data":"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"}
Nov 01 21:21:48.081590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:48.081556    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:48.082622 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:48.082592    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 21:21:48.083074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:48.082776    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 21:21:49.081334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:49.080875    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 21:21:49.081334 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:49.080943    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 21:21:50.078908 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:50.078868    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 21:21:50.079185 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:50.079158    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 21:21:51.081516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:51.080797    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 21:21:51.081516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:51.080858    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 21:21:51.625037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:51.625009    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:51.629008 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:21:51.628976    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:21:52.079011 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:52.078966    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 21:21:52.079333 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:52.079302    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 21:21:53.079566 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:53.079535    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:21:53.080451 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:21:53.080431    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:22:21.628130 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:22:21.628093    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:22:27.480550 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:27.480517    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:22:27.540827 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:27.540801    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:22:51.630812 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:22:51.630774    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:22:54.091349 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:54.091315    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:22:54.091976 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:54.091585    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" podUID="2a254c47-642e-4bdd-adc8-942185944eae" containerName="maas-api" containerID="cri-o://1617c8f08f136962f7b6c724bd045594331ff8c04bd7234b7669ee9296c7b371" gracePeriod=30
Nov 01 21:22:54.190613 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:54.190578    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:22:54.268194 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:54.268157    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:22:55.215178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.215145    5066 generic.go:356] "Generic (PLEG): container finished" podID="2a254c47-642e-4bdd-adc8-942185944eae" containerID="1617c8f08f136962f7b6c724bd045594331ff8c04bd7234b7669ee9296c7b371" exitCode=0
Nov 01 21:22:55.215525 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.215211    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" event={"ID":"2a254c47-642e-4bdd-adc8-942185944eae","Type":"ContainerDied","Data":"1617c8f08f136962f7b6c724bd045594331ff8c04bd7234b7669ee9296c7b371"}
Nov 01 21:22:55.355604 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.355579    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:22:55.444543 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.444436    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\") pod \"2a254c47-642e-4bdd-adc8-942185944eae\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") "
Nov 01 21:22:55.444543 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.444475    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\") pod \"2a254c47-642e-4bdd-adc8-942185944eae\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") "
Nov 01 21:22:55.444756 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.444600    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\") pod \"2a254c47-642e-4bdd-adc8-942185944eae\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") "
Nov 01 21:22:55.444756 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.444629    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") pod \"2a254c47-642e-4bdd-adc8-942185944eae\" (UID: \"2a254c47-642e-4bdd-adc8-942185944eae\") "
Nov 01 21:22:55.444876 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.444819    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca" pod "2a254c47-642e-4bdd-adc8-942185944eae" (UID: "2a254c47-642e-4bdd-adc8-942185944eae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:22:55.446801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.446773    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls" pod "2a254c47-642e-4bdd-adc8-942185944eae" (UID: "2a254c47-642e-4bdd-adc8-942185944eae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:22:55.446801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.446787    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg" pod "2a254c47-642e-4bdd-adc8-942185944eae" (UID: "2a254c47-642e-4bdd-adc8-942185944eae"). InnerVolumeSpecName "kube-api-access-jsntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:22:55.446959 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.446785    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs" pod "2a254c47-642e-4bdd-adc8-942185944eae" (UID: "2a254c47-642e-4bdd-adc8-942185944eae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:22:55.546081 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.546050    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a254c47-642e-4bdd-adc8-942185944eae-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:22:55.546081 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.546077    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:22:55.546081 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.546089    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/2a254c47-642e-4bdd-adc8-942185944eae-kube-api-access-jsntg\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:22:55.546332 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:55.546098    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a254c47-642e-4bdd-adc8-942185944eae-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:22:56.221634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.221599    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g" event={"ID":"2a254c47-642e-4bdd-adc8-942185944eae","Type":"ContainerDied","Data":"5a8b5d9b398c60e3e50dba02f8c033e1e2a99b9d9bf94a0bb2b69090a407a4e1"}
Nov 01 21:22:56.222092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.221655    5066 scope.go:122] "RemoveContainer" containerID="1617c8f08f136962f7b6c724bd045594331ff8c04bd7234b7669ee9296c7b371"
Nov 01 21:22:56.222092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.221613    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"
Nov 01 21:22:56.384447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.384404    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:22:56.434038 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.434001    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-cql7g"]
Nov 01 21:22:56.730351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:56.730319    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a254c47-642e-4bdd-adc8-942185944eae" path="/var/lib/kubelet/pods/2a254c47-642e-4bdd-adc8-942185944eae/volumes"
Nov 01 21:22:59.191561 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:59.191513    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing" containerID="cri-o://e582574b8fc9b62587ea5fc730eee95d7b6a798286767d711ced4fe3074b550a" gracePeriod=25
Nov 01 21:22:59.268760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:59.268717    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing" containerID="cri-o://f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3" gracePeriod=25
Nov 01 21:22:59.706064 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:59.706024    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:22:59.706064 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:22:59.706046    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:23:09.706452 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:09.706415    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:23:09.706855 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:09.706415    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:23:19.349197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.349165    5066 generic.go:356] "Generic (PLEG): container finished" podID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerID="e582574b8fc9b62587ea5fc730eee95d7b6a798286767d711ced4fe3074b550a" exitCode=0
Nov 01 21:23:19.349531 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.349233    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" event={"ID":"a97124b5-3e4c-4051-b70f-7d86f60535ac","Type":"ContainerDied","Data":"e582574b8fc9b62587ea5fc730eee95d7b6a798286767d711ced4fe3074b550a"}
Nov 01 21:23:19.515277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.515250    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:23:19.542046 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.542025    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:23:19.577680 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.577658    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\" (UniqueName: \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\") pod \"a97124b5-3e4c-4051-b70f-7d86f60535ac\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") "
Nov 01 21:23:19.577779 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.577724    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\") pod \"a97124b5-3e4c-4051-b70f-7d86f60535ac\" (UID: \"a97124b5-3e4c-4051-b70f-7d86f60535ac\") "
Nov 01 21:23:19.578051 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.578031    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume" pod "a97124b5-3e4c-4051-b70f-7d86f60535ac" (UID: "a97124b5-3e4c-4051-b70f-7d86f60535ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:23:19.579641 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.579622    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k" pod "a97124b5-3e4c-4051-b70f-7d86f60535ac" (UID: "a97124b5-3e4c-4051-b70f-7d86f60535ac"). InnerVolumeSpecName "kube-api-access-nb24k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:23:19.678615 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.678586    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\") pod \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") "
Nov 01 21:23:19.678785 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.678649    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\" (UniqueName: \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\") pod \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\" (UID: \"84f9fcb8-befe-471c-b81d-5d029b9f6fda\") "
Nov 01 21:23:19.679028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.679000    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume" pod "84f9fcb8-befe-471c-b81d-5d029b9f6fda" (UID: "84f9fcb8-befe-471c-b81d-5d029b9f6fda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:23:19.679091 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.679026    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97124b5-3e4c-4051-b70f-7d86f60535ac-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:19.679091 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.679049    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb24k\" (UniqueName: \"kubernetes.io/projected/a97124b5-3e4c-4051-b70f-7d86f60535ac-kube-api-access-nb24k\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:19.680688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.680668    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr" pod "84f9fcb8-befe-471c-b81d-5d029b9f6fda" (UID: "84f9fcb8-befe-471c-b81d-5d029b9f6fda"). InnerVolumeSpecName "kube-api-access-2q4mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:23:19.779992 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.779922    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4mr\" (UniqueName: \"kubernetes.io/projected/84f9fcb8-befe-471c-b81d-5d029b9f6fda-kube-api-access-2q4mr\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:19.779992 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:19.779946    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f9fcb8-befe-471c-b81d-5d029b9f6fda-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:20.356806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.356780    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"
Nov 01 21:23:20.357261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.356819    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx" event={"ID":"a97124b5-3e4c-4051-b70f-7d86f60535ac","Type":"ContainerDied","Data":"60f2776677ddef32c13ae7016cc271254e6f3bf902ce9c23c4a63cbafc2e3ad7"}
Nov 01 21:23:20.357261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.356889    5066 scope.go:122] "RemoveContainer" containerID="e582574b8fc9b62587ea5fc730eee95d7b6a798286767d711ced4fe3074b550a"
Nov 01 21:23:20.358275 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.358243    5066 generic.go:356] "Generic (PLEG): container finished" podID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerID="f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3" exitCode=0
Nov 01 21:23:20.358275 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.358274    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" event={"ID":"84f9fcb8-befe-471c-b81d-5d029b9f6fda","Type":"ContainerDied","Data":"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"}
Nov 01 21:23:20.358443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.358288    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"
Nov 01 21:23:20.358443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.358306    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8" event={"ID":"84f9fcb8-befe-471c-b81d-5d029b9f6fda","Type":"ContainerDied","Data":"159a589aad5ff00f5bb57830b97018c0757f1e52f2a841802e6d16d76cacc9aa"}
Nov 01 21:23:20.367188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.367171    5066 scope.go:122] "RemoveContainer" containerID="f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"
Nov 01 21:23:20.375367 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.375212    5066 scope.go:122] "RemoveContainer" containerID="f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"
Nov 01 21:23:20.375512 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:23:20.375476    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3\": container with ID starting with f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3 not found: ID does not exist" containerID="f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"
Nov 01 21:23:20.375563 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.375521    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3"} err="failed to get container status \"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3\": rpc error: code = NotFound desc = could not find container \"f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3\": container with ID starting with f3dd15f9edd7ba556c4c487fe57e8b236a6ff9b2fba912cb7699bc0f904babf3 not found: ID does not exist"
Nov 01 21:23:20.519022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.518987    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:23:20.568309 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.568277    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595fr6xh8"]
Nov 01 21:23:20.675920 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.675894    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:23:20.678169 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.678147    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87vbthx"]
Nov 01 21:23:20.730508 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.730471    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" path="/var/lib/kubelet/pods/84f9fcb8-befe-471c-b81d-5d029b9f6fda/volumes"
Nov 01 21:23:20.730859 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:20.730845    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" path="/var/lib/kubelet/pods/a97124b5-3e4c-4051-b70f-7d86f60535ac/volumes"
Nov 01 21:23:21.628215 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:23:21.628173    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:23:23.044036 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:23.043999    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"]
Nov 01 21:23:23.093377 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:23.093343    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4pwm9"]
Nov 01 21:23:24.729694 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:24.729661    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6ada81-0d73-4c30-9c93-4edbcfed78f0" path="/var/lib/kubelet/pods/9a6ada81-0d73-4c30-9c93-4edbcfed78f0/volumes"
Nov 01 21:23:30.649267 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.649231    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"]
Nov 01 21:23:30.650007 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.649979    5066 cpu_manager.go:414] "containerMap: removing container" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing"
Nov 01 21:23:30.650007 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650007    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing"
Nov 01 21:23:30.650186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650050    5066 cpu_manager.go:414] "containerMap: removing container" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing"
Nov 01 21:23:30.650186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650059    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing"
Nov 01 21:23:30.650186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650085    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2a254c47-642e-4bdd-adc8-942185944eae" containerName="maas-api"
Nov 01 21:23:30.650186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650092    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a254c47-642e-4bdd-adc8-942185944eae" containerName="maas-api"
Nov 01 21:23:30.650186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650187    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f9fcb8-befe-471c-b81d-5d029b9f6fda" containerName="payload-pre-processing"
Nov 01 21:23:30.650358 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650201    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97124b5-3e4c-4051-b70f-7d86f60535ac" containerName="payload-processing"
Nov 01 21:23:30.650358 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.650209    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a254c47-642e-4bdd-adc8-942185944eae" containerName="maas-api"
Nov 01 21:23:30.655190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.655165    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.709972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.709933    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"]
Nov 01 21:23:30.755856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.755821    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:23:30.757119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.757096    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:23:30.757226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.757131    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f3eeb7-serving-cert\""
Nov 01 21:23:30.757226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.757164    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:23:30.757226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.757217    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:23:30.791995 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.791955    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.792186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.792021    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwvn\" (UniqueName: \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.792261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.792210    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.792345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.792322    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.893657 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.893618    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.893874 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.893681    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwvn\" (UniqueName: \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.893874 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.893726    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.893999 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.893901    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.894062 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:23:30.894039    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-serving-cert: secret "maas-api-e2e-worker-single-main-f3eeb7-serving-cert" not found
Nov 01 21:23:30.894167 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:23:30.894152    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls podName:021a3e34-5eb2-4c22-8d2d-b2f40b2a4107 nodeName:}" failed. No retries permitted until 2026-11-01 21:23:31.394099468 +0000 UTC m=+4864.778997036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls") pod "maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" (UID: "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107") : secret "maas-api-e2e-worker-single-main-f3eeb7-serving-cert" not found
Nov 01 21:23:30.894654 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.894605    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.910055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.909972    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:30.950856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:30.950826    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwvn\" (UniqueName: \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:31.399649 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:31.399611    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:31.402207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:31.402176    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:31.568376 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:31.568343    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:31.947022 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:23:31.946983    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021a3e34_5eb2_4c22_8d2d_b2f40b2a4107.slice/crio-15104f9938bf482c12d34f0b0c712e6bb905b2c062cb6065d174dc76fc2fc5ff WatchSource:0}: Error finding container 15104f9938bf482c12d34f0b0c712e6bb905b2c062cb6065d174dc76fc2fc5ff: Status 404 returned error can't find the container with id 15104f9938bf482c12d34f0b0c712e6bb905b2c062cb6065d174dc76fc2fc5ff
Nov 01 21:23:31.948123 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:31.948091    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"]
Nov 01 21:23:32.138359 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.138330    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:23:32.143638 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.143614    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.197318 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.197235    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f3eeb7\""
Nov 01 21:23:32.222516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.215526    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:23:32.245103 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.245063    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-469613a7-dockercfg-m68hr\""
Nov 01 21:23:32.295570 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.295538    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:23:32.299211 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.299191    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.314920 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.314899    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.315017 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.314950    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl44g\" (UniqueName: \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.355371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.355347    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:23:32.415717 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.415682    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.415889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.415771    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.415889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.415807    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl44g\" (UniqueName: \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.415889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.415841    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglkc\" (UniqueName: \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.416452 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.416427    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.435621 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.435593    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" event={"ID":"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107","Type":"ContainerStarted","Data":"15104f9938bf482c12d34f0b0c712e6bb905b2c062cb6065d174dc76fc2fc5ff"}
Nov 01 21:23:32.472351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.472320    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl44g\" (UniqueName: \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\") pod \"payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.517154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.517128    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.517288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.517167    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglkc\" (UniqueName: \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.517838 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.517819    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.573416 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.573386    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglkc\" (UniqueName: \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\") pod \"payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.618340 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.618318    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:32.757753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.757658    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:32.996265 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:32.996239    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:23:32.997810 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:23:32.997782    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf75a1_9a37_4507_8907_46a02fabef8f.slice/crio-77f2c9effed203d1d04a29bed1ac572c340e95f6c5b2c1061c12c39bbfdb1578 WatchSource:0}: Error finding container 77f2c9effed203d1d04a29bed1ac572c340e95f6c5b2c1061c12c39bbfdb1578: Status 404 returned error can't find the container with id 77f2c9effed203d1d04a29bed1ac572c340e95f6c5b2c1061c12c39bbfdb1578
Nov 01 21:23:33.129693 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.129670    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:23:33.131278 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:23:33.131246    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38768d8b_3092_4593_834e_ef356a56a460.slice/crio-fc9df0039b8bec11ac6a572d0a549d3706acfaedb5180159f1ff0b0b125bcfc4 WatchSource:0}: Error finding container fc9df0039b8bec11ac6a572d0a549d3706acfaedb5180159f1ff0b0b125bcfc4: Status 404 returned error can't find the container with id fc9df0039b8bec11ac6a572d0a549d3706acfaedb5180159f1ff0b0b125bcfc4
Nov 01 21:23:33.443732 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.443649    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" event={"ID":"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107","Type":"ContainerStarted","Data":"e8b8af7aeb83f323200b2ffaafad76d27df9cc095a0caead38ff6bdad783b4bd"}
Nov 01 21:23:33.443899 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.443752    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:33.445285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.445262    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" event={"ID":"00bf75a1-9a37-4507-8907-46a02fabef8f","Type":"ContainerStarted","Data":"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"}
Nov 01 21:23:33.445407 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.445297    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" event={"ID":"00bf75a1-9a37-4507-8907-46a02fabef8f","Type":"ContainerStarted","Data":"77f2c9effed203d1d04a29bed1ac572c340e95f6c5b2c1061c12c39bbfdb1578"}
Nov 01 21:23:33.445407 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.445319    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:33.447564 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.447541    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" event={"ID":"38768d8b-3092-4593-834e-ef356a56a460","Type":"ContainerStarted","Data":"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"}
Nov 01 21:23:33.447656 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.447577    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" event={"ID":"38768d8b-3092-4593-834e-ef356a56a460","Type":"ContainerStarted","Data":"fc9df0039b8bec11ac6a572d0a549d3706acfaedb5180159f1ff0b0b125bcfc4"}
Nov 01 21:23:33.447703 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.447675    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:33.556769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.556716    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" podStartSLOduration=2.65779861 podStartE2EDuration="3.556701189s" totalImagesPullingTime="898.902579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:23:33.555458551 +0000 UTC m=+4866.940356324" watchObservedRunningTime="2026-11-01 21:23:33.556701189 +0000 UTC m=+4866.941598778"
Nov 01 21:23:33.670452 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.670381    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" podStartSLOduration=1.670362961 podStartE2EDuration="1.670362961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:23:33.669574154 +0000 UTC m=+4867.054471744" watchObservedRunningTime="2026-11-01 21:23:33.670362961 +0000 UTC m=+4867.055260551"
Nov 01 21:23:33.795401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:33.795338    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" podStartSLOduration=1.795317246 podStartE2EDuration="1.795317246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:23:33.780772794 +0000 UTC m=+4867.165670384" watchObservedRunningTime="2026-11-01 21:23:33.795317246 +0000 UTC m=+4867.180214837"
Nov 01 21:23:39.459796 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:39.459765    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:44.457372 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:44.457339    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:23:44.457803 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:44.457417    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:23:51.627782 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:23:51.627744    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:23:55.812390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:55.812356    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:23:55.917642 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:55.917610    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:23:58.411750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.411720    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"]
Nov 01 21:23:58.412278 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.411905    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" podUID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" containerName="maas-api" containerID="cri-o://e8b8af7aeb83f323200b2ffaafad76d27df9cc095a0caead38ff6bdad783b4bd" gracePeriod=30
Nov 01 21:23:58.598467 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.598431    5066 generic.go:356] "Generic (PLEG): container finished" podID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" containerID="e8b8af7aeb83f323200b2ffaafad76d27df9cc095a0caead38ff6bdad783b4bd" exitCode=0
Nov 01 21:23:58.598641 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.598483    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" event={"ID":"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107","Type":"ContainerDied","Data":"e8b8af7aeb83f323200b2ffaafad76d27df9cc095a0caead38ff6bdad783b4bd"}
Nov 01 21:23:58.671452 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.671384    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:58.777022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.776991    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\") pod \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") "
Nov 01 21:23:58.777022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.777023    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") pod \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") "
Nov 01 21:23:58.777235 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.777082    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\" (UniqueName: \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\") pod \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") "
Nov 01 21:23:58.777235 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.777153    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\") pod \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\" (UID: \"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107\") "
Nov 01 21:23:58.777629 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.777596    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca" pod "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" (UID: "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:23:58.779247 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.779218    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn" pod "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" (UID: "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107"). InnerVolumeSpecName "kube-api-access-mqwvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:23:58.779354 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.779245    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs" pod "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" (UID: "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:23:58.779354 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.779266    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls" pod "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" (UID: "021a3e34-5eb2-4c22-8d2d-b2f40b2a4107"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:23:58.878020 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.877981    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqwvn\" (UniqueName: \"kubernetes.io/projected/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-kube-api-access-mqwvn\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:58.878020 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.878020    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:58.878227 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.878034    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:58.878227 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:58.878048    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:23:59.606369 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:59.606330    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"
Nov 01 21:23:59.606869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:59.606371    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt" event={"ID":"021a3e34-5eb2-4c22-8d2d-b2f40b2a4107","Type":"ContainerDied","Data":"15104f9938bf482c12d34f0b0c712e6bb905b2c062cb6065d174dc76fc2fc5ff"}
Nov 01 21:23:59.606869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:23:59.606431    5066 scope.go:122] "RemoveContainer" containerID="e8b8af7aeb83f323200b2ffaafad76d27df9cc095a0caead38ff6bdad783b4bd"
Nov 01 21:24:00.812828 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:00.812769    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" podUID="00bf75a1-9a37-4507-8907-46a02fabef8f" containerName="payload-processing" containerID="cri-o://f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292" gracePeriod=25
Nov 01 21:24:00.917899 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:00.917864    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" podUID="38768d8b-3092-4593-834e-ef356a56a460" containerName="payload-pre-processing" containerID="cri-o://37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9" gracePeriod=25
Nov 01 21:24:01.099895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.099868    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:24:01.171949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.171927    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:24:01.226777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.226744    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\") pod \"00bf75a1-9a37-4507-8907-46a02fabef8f\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") "
Nov 01 21:24:01.226927 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.226817    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\" (UniqueName: \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\") pod \"00bf75a1-9a37-4507-8907-46a02fabef8f\" (UID: \"00bf75a1-9a37-4507-8907-46a02fabef8f\") "
Nov 01 21:24:01.227097 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.227073    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume" pod "00bf75a1-9a37-4507-8907-46a02fabef8f" (UID: "00bf75a1-9a37-4507-8907-46a02fabef8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:24:01.227201 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.227183    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf75a1-9a37-4507-8907-46a02fabef8f-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:01.228804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.228781    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc" pod "00bf75a1-9a37-4507-8907-46a02fabef8f" (UID: "00bf75a1-9a37-4507-8907-46a02fabef8f"). InnerVolumeSpecName "kube-api-access-qglkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:24:01.327735 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.327660    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\") pod \"38768d8b-3092-4593-834e-ef356a56a460\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") "
Nov 01 21:24:01.327865 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.327825    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\" (UniqueName: \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\") pod \"38768d8b-3092-4593-834e-ef356a56a460\" (UID: \"38768d8b-3092-4593-834e-ef356a56a460\") "
Nov 01 21:24:01.328101 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.328080    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume" pod "38768d8b-3092-4593-834e-ef356a56a460" (UID: "38768d8b-3092-4593-834e-ef356a56a460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:24:01.328207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.328192    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglkc\" (UniqueName: \"kubernetes.io/projected/00bf75a1-9a37-4507-8907-46a02fabef8f-kube-api-access-qglkc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:01.328243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.328211    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38768d8b-3092-4593-834e-ef356a56a460-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:01.329770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.329749    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g" pod "38768d8b-3092-4593-834e-ef356a56a460" (UID: "38768d8b-3092-4593-834e-ef356a56a460"). InnerVolumeSpecName "kube-api-access-gl44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:24:01.428881 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.428853    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl44g\" (UniqueName: \"kubernetes.io/projected/38768d8b-3092-4593-834e-ef356a56a460-kube-api-access-gl44g\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:01.621875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.621785    5066 generic.go:356] "Generic (PLEG): container finished" podID="00bf75a1-9a37-4507-8907-46a02fabef8f" containerID="f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292" exitCode=0
Nov 01 21:24:01.621875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.621832    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"
Nov 01 21:24:01.621875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.621865    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" event={"ID":"00bf75a1-9a37-4507-8907-46a02fabef8f","Type":"ContainerDied","Data":"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"}
Nov 01 21:24:01.622166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.621921    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk" event={"ID":"00bf75a1-9a37-4507-8907-46a02fabef8f","Type":"ContainerDied","Data":"77f2c9effed203d1d04a29bed1ac572c340e95f6c5b2c1061c12c39bbfdb1578"}
Nov 01 21:24:01.622166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.621949    5066 scope.go:122] "RemoveContainer" containerID="f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"
Nov 01 21:24:01.624190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.623558    5066 generic.go:356] "Generic (PLEG): container finished" podID="38768d8b-3092-4593-834e-ef356a56a460" containerID="37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9" exitCode=0
Nov 01 21:24:01.624190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.623627    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"
Nov 01 21:24:01.624190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.624106    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" event={"ID":"38768d8b-3092-4593-834e-ef356a56a460","Type":"ContainerDied","Data":"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"}
Nov 01 21:24:01.624190 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.624156    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc" event={"ID":"38768d8b-3092-4593-834e-ef356a56a460","Type":"ContainerDied","Data":"fc9df0039b8bec11ac6a572d0a549d3706acfaedb5180159f1ff0b0b125bcfc4"}
Nov 01 21:24:01.633734 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.633646    5066 scope.go:122] "RemoveContainer" containerID="f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"
Nov 01 21:24:01.634036 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:01.633962    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292\": container with ID starting with f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292 not found: ID does not exist" containerID="f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"
Nov 01 21:24:01.634036 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.633989    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292"} err="failed to get container status \"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292\": rpc error: code = NotFound desc = could not find container \"f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292\": container with ID starting with f436237841404057e5337ebd5dd5b8e6406ebd21f789c582d7bee888d2916292 not found: ID does not exist"
Nov 01 21:24:01.634036 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.634005    5066 scope.go:122] "RemoveContainer" containerID="37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"
Nov 01 21:24:01.642189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.642169    5066 scope.go:122] "RemoveContainer" containerID="37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"
Nov 01 21:24:01.642479 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:01.642458    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9\": container with ID starting with 37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9 not found: ID does not exist" containerID="37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"
Nov 01 21:24:01.642597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:01.642482    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9"} err="failed to get container status \"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9\": rpc error: code = NotFound desc = could not find container \"37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9\": container with ID starting with 37c646c055d04786f485ace59d7106bf92d9438509e36a222e875fdc702d0ef9 not found: ID does not exist"
Nov 01 21:24:13.919407 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:13.919323    5066 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-223.us-west-1.compute.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:24:21.628991 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:21.628948    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:24:23.919893 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:23.919624    5066 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-223.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:24:28.174794 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:28.174759    5066 scope.go:122] "RemoveContainer" containerID="ba9afdebd9fe01a62cb889dedabed2c763264b78d7b7323e2366f3abab91a5bf"
Nov 01 21:24:33.920578 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:33.920533    5066 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-223.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:24:43.921727 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:43.921690    5066 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-223.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:24:44.661221 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661186    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661237    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mr6jl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661300    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bthm5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661312    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661356    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f3eeb7-openshift-default-dockercfg-nn4lb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661368    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661380    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661396    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661402    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661447    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nw87d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661446    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661478    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661509    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661538    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661519    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661550    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661577    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661578    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661605    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661609    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661630    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661628    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661659    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661667    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661689    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661717    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661741    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dmdkt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661748    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661772    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661784    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661772    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661805    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661821    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661837    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661843    5066 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661870    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4jvr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661873    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7gfgj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661875    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661899    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661912    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661921    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.661924 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661936    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661963    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661965    5066 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661968    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.661955    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662000    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662003    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662001    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f22l2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662020    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662041    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662046    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662068    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662076    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bwz74\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662082    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662093    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662142    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662152    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662215    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662250    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662256    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p45lr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662273    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662281    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662294    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662341    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662366    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662379    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662387    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662349    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662381    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ftcsj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662395    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662399    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-wskl6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662414    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662424    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662427    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662432    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662437    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662410    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662436    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662451    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662440    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662467    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4s8m4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662474    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4n5t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662380    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662506    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662460    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662487    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662519    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662525    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-247ns\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662512    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662543    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662547    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662519    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662555    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662572    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662576    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jjkzl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662586    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662589    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662593    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662600    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662604    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662597    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662616    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hwgb7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662619    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662624    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662628    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662556    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662555    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662572    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662629    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662640    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662648    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662647    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662661    5066 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662666    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662677    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662676    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662687    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662694    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662702    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662711    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wmgh6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662715    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662717    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662727    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662706    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662744    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cc7fv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662752    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662759    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662768    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662788    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662803    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kcmjb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:44.662801    5066 controller.go:251] "Failed to update lease" err="Put \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-223.us-west-1.compute.internal?timeout=10s\": http2: client connection lost"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662794    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662816    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662824    5066 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662847    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662850    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662867    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662875    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662826    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662852    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662903    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662908    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662912    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662910    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662914    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fqldw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662932    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662941    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662943    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662949    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662938    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662975    5066 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662979    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662987    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.662986    5066 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663000    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663015    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663017    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663021    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-96lsv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663035    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2mhc5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663045    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663050    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-np7zt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663051    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663062    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663076    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:44.663404 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:44.663070    5066 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 01 21:24:45.515650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.515617    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.516504 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.516472    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mr6jl\""
Nov 01 21:24:45.566386 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.566359    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.566523 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.566447    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.567230 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567212    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-247ns\""
Nov 01 21:24:45.567316 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567300    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 21:24:45.567370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567355    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 21:24:45.567413 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567402    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f3eeb7-openshift-default-dockercfg-nn4lb\""
Nov 01 21:24:45.567450 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567415    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 21:24:45.567450 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.567437    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.617470 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617447    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 21:24:45.617652 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617488    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 21:24:45.617759 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617549    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 21:24:45.617759 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617562    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.617910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617603    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 21:24:45.617910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617603    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.617910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617607    5066 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:24:45.617910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617659    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.617910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.617707    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 21:24:45.618295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.618258    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\""
Nov 01 21:24:45.618390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.618296    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ftcsj\""
Nov 01 21:24:45.618472 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.618451    5066 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:24:45.618771 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.618598    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 21:24:45.623379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.623358    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\""
Nov 01 21:24:45.668328 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.668298    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 21:24:45.668549 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.668385    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\""
Nov 01 21:24:45.668676 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.668661    5066 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:24:45.669158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669142    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.669242 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669219    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 21:24:45.669312 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669255    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cc7fv\""
Nov 01 21:24:45.669312 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669303    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bthm5\""
Nov 01 21:24:45.669395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669305    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.669395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669331    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.669395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.669375    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.674238 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.674218    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 21:24:45.674353 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.674275    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 21:24:45.685563 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.685547    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 21:24:45.718902 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.718883    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 21:24:45.719215 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719200    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\""
Nov 01 21:24:45.719330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719311    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.719433 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719416    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 21:24:45.719469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719444    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 21:24:45.719469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719453    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\""
Nov 01 21:24:45.719606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719595    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\""
Nov 01 21:24:45.719686 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.719673    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.727018 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.727001    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 21:24:45.732236 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.732218    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.732330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.732304    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 21:24:45.738222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.738206    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7gfgj\""
Nov 01 21:24:45.740200 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.740185    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 21:24:45.751736 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.751716    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 21:24:45.752636 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.752622    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\""
Nov 01 21:24:45.757368 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.757354    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\""
Nov 01 21:24:45.760155 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.760141    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 21:24:45.766239 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.766189    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 21:24:45.773216 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.773198    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.782130 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.782113    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-np7zt\""
Nov 01 21:24:45.783546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.783531    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.783854 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.783840    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p45lr\""
Nov 01 21:24:45.786889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.786877    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.787182 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.787169    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.788217 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.788205    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.788272 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.788233    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4jvr\""
Nov 01 21:24:45.788366 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.788356    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 21:24:45.793726 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.793712    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.817108 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.817086    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.817212 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.817167    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 21:24:45.817270 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.817211    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 21:24:45.817560 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.817544    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 21:24:45.847558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847535    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.847558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847541    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 21:24:45.847691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847570    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 21:24:45.847691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847600    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 21:24:45.847760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847720    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.847793 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.847759    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 21:24:45.849481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.849467    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 21:24:45.851658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.851644    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.856544 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.856530    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 21:24:45.856590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.856574    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wskl6\""
Nov 01 21:24:45.865749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.865731    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.865854 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.865842    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dmdkt\""
Nov 01 21:24:45.872661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.872641    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.880912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.880896    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wmgh6\""
Nov 01 21:24:45.887406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.887391    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.889829 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.889812    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\""
Nov 01 21:24:45.890347 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.890335    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-96lsv\""
Nov 01 21:24:45.894217 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.894199    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\""
Nov 01 21:24:45.907884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.907856    5066 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 21:24:45.912128 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.912103    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f3eeb7-75cdfb5b8b-mbhlt"]
Nov 01 21:24:45.912226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.912146    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:24:45.912358 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.912337    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy" containerID="cri-o://c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4" gracePeriod=30
Nov 01 21:24:45.931794 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.929936    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.931794 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.930133    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4s8m4\""
Nov 01 21:24:45.934691 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.934670    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.935760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.935743    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 21:24:45.938283 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.938257    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fqldw\""
Nov 01 21:24:45.940632 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.940609    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 21:24:45.954384 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.954366    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 21:24:45.961414 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.961399    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4n5t\""
Nov 01 21:24:45.962476 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.962462    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.965115 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.965100    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 21:24:45.977501 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.977471    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:24:45.980713 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.980693    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f3eeb7-5fbc68f87mt8jk"]
Nov 01 21:24:45.987251 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.987236    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 21:24:45.987307 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.987256    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 21:24:45.990767 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:45.990752    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.017697 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.017635    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.019715 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.019699    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 21:24:46.020885 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.020870    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 21:24:46.020957 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.020946    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 21:24:46.025217 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.025201    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.025319 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.025267    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 21:24:46.027821 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.027806    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.029981 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.029966    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l79xf\""
Nov 01 21:24:46.033056 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.033038    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\""
Nov 01 21:24:46.039735 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.039710    5066 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:24:46.040811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.040793    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 21:24:46.046542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.046527    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bwz74\""
Nov 01 21:24:46.047877 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.047862    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 21:24:46.051123 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.051110    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\""
Nov 01 21:24:46.056119 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.056100    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kcmjb\""
Nov 01 21:24:46.058473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.058458    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 21:24:46.072168 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.072153    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\""
Nov 01 21:24:46.076683 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.076668    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.083747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.083730    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 21:24:46.090427 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.090312    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.091825 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.091809    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.097210 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.097188    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 21:24:46.097990 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.097970    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.107604 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.107582    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 21:24:46.116112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.116091    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 21:24:46.116223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.116163    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.121187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.121167    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.121371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.121353    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.127880 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.127861    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 21:24:46.133655 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.133637    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 21:24:46.135037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.135021    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 21:24:46.138524 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.138484    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 21:24:46.150594 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.150565    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:24:46.153418 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.153398    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f3eeb7-6595f762bc"]
Nov 01 21:24:46.155808 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.155793    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hwgb7\""
Nov 01 21:24:46.159266 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.159249    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.166919 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.166904    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.171222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.171208    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 21:24:46.174398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.174382    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.176529 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.176514    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 21:24:46.176585 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.176562    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 21:24:46.188835 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.188817    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\""
Nov 01 21:24:46.195603 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.195584    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 21:24:46.196198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.196184    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.204391 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.204374    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 21:24:46.209464 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.209445    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f22l2\""
Nov 01 21:24:46.218038 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.218019    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 21:24:46.218129 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.218085    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 21:24:46.220466 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.220454    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.237569 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.237553    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nw87d\""
Nov 01 21:24:46.241502 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.241483    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jjkzl\""
Nov 01 21:24:46.244646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.244631    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\""
Nov 01 21:24:46.255993 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.255968    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.256101 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.256048    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.256842 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.256827    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\""
Nov 01 21:24:46.266248 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.266233    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\""
Nov 01 21:24:46.269862 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.269784    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 21:24:46.278205 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.278189    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.286336 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.286319    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 21:24:46.291158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.291141    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 21:24:46.298281 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.298266    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2mhc5\""
Nov 01 21:24:46.303624 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.303608    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.307486 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.307472    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 21:24:46.310469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.310457    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 21:24:46.310580 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.310569    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 21:24:46.731433 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.731394    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00bf75a1-9a37-4507-8907-46a02fabef8f" path="/var/lib/kubelet/pods/00bf75a1-9a37-4507-8907-46a02fabef8f/volumes"
Nov 01 21:24:46.732022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.732001    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" path="/var/lib/kubelet/pods/021a3e34-5eb2-4c22-8d2d-b2f40b2a4107/volumes"
Nov 01 21:24:46.732407 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:46.732389    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38768d8b-3092-4593-834e-ef356a56a460" path="/var/lib/kubelet/pods/38768d8b-3092-4593-834e-ef356a56a460/volumes"
Nov 01 21:24:51.227589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.227548    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:24:51.299005 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.298940    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299059    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299092    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299151    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299222 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299188    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299253    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299300    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299341    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299430    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\") pod \"0a8fb376-4d54-4702-a66c-c227ac78a694\" (UID: \"0a8fb376-4d54-4702-a66c-c227ac78a694\") "
Nov 01 21:24:51.299946 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.299825    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:24:51.300727 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.300214    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:24:51.300727 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.300333    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:24:51.300727 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.300392    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:24:51.300946 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.300746    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:24:51.302635 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.302603    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:24:51.302866 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.302839    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "kube-api-access-mqsw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:24:51.304201 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.304174    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:24:51.304798 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.304770    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy" pod "0a8fb376-4d54-4702-a66c-c227ac78a694" (UID: "0a8fb376-4d54-4702-a66c-c227ac78a694"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401214    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401252    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8fb376-4d54-4702-a66c-c227ac78a694-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401269    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401283    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401297    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401311    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401324    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401339    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8fb376-4d54-4702-a66c-c227ac78a694-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.401381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.401353    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsw5\" (UniqueName: \"kubernetes.io/projected/0a8fb376-4d54-4702-a66c-c227ac78a694-kube-api-access-mqsw5\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:24:51.646177 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:51.646135    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:24:51.942295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.942194    5066 generic.go:356] "Generic (PLEG): container finished" podID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerID="c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4" exitCode=0
Nov 01 21:24:51.942469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.942323    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerDied","Data":"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"}
Nov 01 21:24:51.942469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.942379    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v" event={"ID":"0a8fb376-4d54-4702-a66c-c227ac78a694","Type":"ContainerDied","Data":"157f0f1c80ffc628f96f235777d1d689909ecbe909eec2a46701a04a7a799311"}
Nov 01 21:24:51.942469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.942410    5066 scope.go:122] "RemoveContainer" containerID="c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"
Nov 01 21:24:51.942683 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.942602    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"
Nov 01 21:24:51.959092 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.959067    5066 scope.go:122] "RemoveContainer" containerID="cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"
Nov 01 21:24:51.988312 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.988266    5066 scope.go:122] "RemoveContainer" containerID="c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"
Nov 01 21:24:51.988786 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:51.988715    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4\": container with ID starting with c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4 not found: ID does not exist" containerID="c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"
Nov 01 21:24:51.988786 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.988760    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4"} err="failed to get container status \"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4\": rpc error: code = NotFound desc = could not find container \"c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4\": container with ID starting with c1b49c2b12aa77db9e00a2f96d8751a6d71f40ade33f56b0a6781d19b1a273e4 not found: ID does not exist"
Nov 01 21:24:51.988786 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.988789    5066 scope.go:122] "RemoveContainer" containerID="cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"
Nov 01 21:24:51.989853 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:24:51.989475    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a\": container with ID starting with cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a not found: ID does not exist" containerID="cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"
Nov 01 21:24:51.989975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:51.989868    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a"} err="failed to get container status \"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a\": rpc error: code = NotFound desc = could not find container \"cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a\": container with ID starting with cb1da38319fe5a63ff59268dc8db545f927fff35f39211ec10c979782d48638a not found: ID does not exist"
Nov 01 21:24:52.189518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:52.178024    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:24:52.211573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:52.196520    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f3eeb7-openshift-default-697bf4ccc7hsg6v"]
Nov 01 21:24:52.748010 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:24:52.747940    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" path="/var/lib/kubelet/pods/0a8fb376-4d54-4702-a66c-c227ac78a694/volumes"
Nov 01 21:25:08.551937 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.551884    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"]
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552368    5066 cpu_manager.go:414] "containerMap: removing container" podUID="00bf75a1-9a37-4507-8907-46a02fabef8f" containerName="payload-processing"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552381    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bf75a1-9a37-4507-8907-46a02fabef8f" containerName="payload-processing"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552402    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552407    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552424    5066 cpu_manager.go:414] "containerMap: removing container" podUID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" containerName="maas-api"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552429    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" containerName="maas-api"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552440    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552445    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552453    5066 cpu_manager.go:414] "containerMap: removing container" podUID="38768d8b-3092-4593-834e-ef356a56a460" containerName="payload-pre-processing"
Nov 01 21:25:08.552469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552458    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38768d8b-3092-4593-834e-ef356a56a460" containerName="payload-pre-processing"
Nov 01 21:25:08.552795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552552    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552563    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="021a3e34-5eb2-4c22-8d2d-b2f40b2a4107" containerName="maas-api"
Nov 01 21:25:08.552795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552571    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8fb376-4d54-4702-a66c-c227ac78a694" containerName="istio-proxy"
Nov 01 21:25:08.552795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552576    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="38768d8b-3092-4593-834e-ef356a56a460" containerName="payload-pre-processing"
Nov 01 21:25:08.552795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.552584    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bf75a1-9a37-4507-8907-46a02fabef8f" containerName="payload-processing"
Nov 01 21:25:08.557597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.557576    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.609336 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.609303    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"]
Nov 01 21:25:08.655535 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.655486    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:25:08.655535 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.655508    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:25:08.655535 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.655528    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\""
Nov 01 21:25:08.655763 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.655547    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:25:08.666458 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.666427    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.666619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.666575    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hp9\" (UniqueName: \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.666668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.666623    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.767754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.767716    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.767956 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.767785    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.767956 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.767896    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hp9\" (UniqueName: \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.768409 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.768389    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.770160 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.770138    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.825316 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.825239    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hp9\" (UniqueName: \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\") pod \"maas-api-revoke-sub-452b6d7b-xwvr7\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:08.874220 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:08.874172    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:09.249769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:09.249731    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"]
Nov 01 21:25:09.249907 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:25:09.249791    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77d2c20_ceb6_43b1_a65e_912e93f74e3c.slice/crio-60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88 WatchSource:0}: Error finding container 60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88: Status 404 returned error can't find the container with id 60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88
Nov 01 21:25:10.103864 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:10.103827    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7" event={"ID":"c77d2c20-ceb6-43b1-a65e-912e93f74e3c","Type":"ContainerStarted","Data":"b62be6735888e17fd5483896066b05fb990bbfb66bab00635ce164d7681be284"}
Nov 01 21:25:10.104306 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:10.103888    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7" event={"ID":"c77d2c20-ceb6-43b1-a65e-912e93f74e3c","Type":"ContainerStarted","Data":"60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88"}
Nov 01 21:25:10.219153 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:10.219096    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7" podStartSLOduration=2.219081046 podStartE2EDuration="2.219081046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:10.217195042 +0000 UTC m=+4963.602092633" watchObservedRunningTime="2026-11-01 21:25:10.219081046 +0000 UTC m=+4963.603978636"
Nov 01 21:25:11.114698 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:11.114656    5066 generic.go:356] "Generic (PLEG): container finished" podID="c77d2c20-ceb6-43b1-a65e-912e93f74e3c" containerID="b62be6735888e17fd5483896066b05fb990bbfb66bab00635ce164d7681be284" exitCode=0
Nov 01 21:25:11.115398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:11.114726    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7" event={"ID":"c77d2c20-ceb6-43b1-a65e-912e93f74e3c","Type":"ContainerDied","Data":"b62be6735888e17fd5483896066b05fb990bbfb66bab00635ce164d7681be284"}
Nov 01 21:25:12.945765 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:12.945725    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:13.141584 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.140124    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\") pod \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") "
Nov 01 21:25:13.141584 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.140272    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\" (UniqueName: \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\") pod \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") "
Nov 01 21:25:13.141584 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.140322    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\") pod \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\" (UID: \"c77d2c20-ceb6-43b1-a65e-912e93f74e3c\") "
Nov 01 21:25:13.159661 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.159617    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca" pod "c77d2c20-ceb6-43b1-a65e-912e93f74e3c" (UID: "c77d2c20-ceb6-43b1-a65e-912e93f74e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:25:13.195434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.195119    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7" event={"ID":"c77d2c20-ceb6-43b1-a65e-912e93f74e3c","Type":"ContainerDied","Data":"60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88"}
Nov 01 21:25:13.195434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.195189    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a8a4b98b96ab96817e616d72f444917232e495c02cc32558c769e7625acf88"
Nov 01 21:25:13.195434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.195270    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"
Nov 01 21:25:13.208727 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.208687    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9" pod "c77d2c20-ceb6-43b1-a65e-912e93f74e3c" (UID: "c77d2c20-ceb6-43b1-a65e-912e93f74e3c"). InnerVolumeSpecName "kube-api-access-h2hp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:25:13.258169 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.243186    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2hp9\" (UniqueName: \"kubernetes.io/projected/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-kube-api-access-h2hp9\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:13.258169 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.258092    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-openshift-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:13.258169 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.250839    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca" pod "c77d2c20-ceb6-43b1-a65e-912e93f74e3c" (UID: "c77d2c20-ceb6-43b1-a65e-912e93f74e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:25:13.378519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:13.376218    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77d2c20-ceb6-43b1-a65e-912e93f74e3c-maas-api-xks-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:21.632056 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:25:21.632016    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:25:33.846449 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.846415    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"]
Nov 01 21:25:33.846933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.846885    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c77d2c20-ceb6-43b1-a65e-912e93f74e3c" containerName="revoke-keys"
Nov 01 21:25:33.846933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.846898    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77d2c20-ceb6-43b1-a65e-912e93f74e3c" containerName="revoke-keys"
Nov 01 21:25:33.847013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.846979    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77d2c20-ceb6-43b1-a65e-912e93f74e3c" containerName="revoke-keys"
Nov 01 21:25:33.851988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.851960    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:33.906285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.906249    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"]
Nov 01 21:25:33.952690 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.952654    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:25:33.952903 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.952883    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:25:33.952976 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.952895    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k82j5\""
Nov 01 21:25:33.955542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:33.955524    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:25:34.030973 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.030937    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.031178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.031093    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.031178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.031151    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk446\" (UniqueName: \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.132193 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.132101    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.132362 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.132192    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.132362 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.132218    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk446\" (UniqueName: \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.132953 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.132932    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.134401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.134378    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.189676 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.189642    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk446\" (UniqueName: \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\") pod \"maas-api-revoke-sub-3e31a506-cmf8z\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.465289 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.465245    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:34.893774 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:34.893263    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"]
Nov 01 21:25:35.426911 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:35.426738    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z" event={"ID":"89525176-b0ec-468b-b745-2fb9232b9971","Type":"ContainerStarted","Data":"17e3ad0f517c6117b365660690bfc96b670e4fabb975c13f95d68b834ccc6916"}
Nov 01 21:25:35.426911 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:35.426811    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z" event={"ID":"89525176-b0ec-468b-b745-2fb9232b9971","Type":"ContainerStarted","Data":"25a7677a8b1d9e0e688aa2dbd49b23538ccb46b09e7673a881b96497a22f37e3"}
Nov 01 21:25:35.545527 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:35.543899    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z" podStartSLOduration=2.543879317 podStartE2EDuration="2.543879317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:25:35.54288252 +0000 UTC m=+4988.927780124" watchObservedRunningTime="2026-11-01 21:25:35.543879317 +0000 UTC m=+4988.928776921"
Nov 01 21:25:36.447100 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:36.447007    5066 generic.go:356] "Generic (PLEG): container finished" podID="89525176-b0ec-468b-b745-2fb9232b9971" containerID="17e3ad0f517c6117b365660690bfc96b670e4fabb975c13f95d68b834ccc6916" exitCode=0
Nov 01 21:25:36.447842 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:36.447757    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z" event={"ID":"89525176-b0ec-468b-b745-2fb9232b9971","Type":"ContainerDied","Data":"17e3ad0f517c6117b365660690bfc96b670e4fabb975c13f95d68b834ccc6916"}
Nov 01 21:25:38.129189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.128809    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:38.214900 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.214850    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\") pod \"89525176-b0ec-468b-b745-2fb9232b9971\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") "
Nov 01 21:25:38.215208 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.215185    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\") pod \"89525176-b0ec-468b-b745-2fb9232b9971\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") "
Nov 01 21:25:38.215398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.215381    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\" (UniqueName: \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\") pod \"89525176-b0ec-468b-b745-2fb9232b9971\" (UID: \"89525176-b0ec-468b-b745-2fb9232b9971\") "
Nov 01 21:25:38.232743 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.232704    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446" pod "89525176-b0ec-468b-b745-2fb9232b9971" (UID: "89525176-b0ec-468b-b745-2fb9232b9971"). InnerVolumeSpecName "kube-api-access-jk446". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:25:38.321484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.321356    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk446\" (UniqueName: \"kubernetes.io/projected/89525176-b0ec-468b-b745-2fb9232b9971-kube-api-access-jk446\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:38.348528 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.326531    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca" pod "89525176-b0ec-468b-b745-2fb9232b9971" (UID: "89525176-b0ec-468b-b745-2fb9232b9971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:25:38.348528 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.326624    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca" pod "89525176-b0ec-468b-b745-2fb9232b9971" (UID: "89525176-b0ec-468b-b745-2fb9232b9971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:25:38.423562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.423526    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-xks-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:38.423795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.423775    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89525176-b0ec-468b-b745-2fb9232b9971-maas-api-openshift-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:25:38.475700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.475655    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z" event={"ID":"89525176-b0ec-468b-b745-2fb9232b9971","Type":"ContainerDied","Data":"25a7677a8b1d9e0e688aa2dbd49b23538ccb46b09e7673a881b96497a22f37e3"}
Nov 01 21:25:38.475906 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.475753    5066 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a7677a8b1d9e0e688aa2dbd49b23538ccb46b09e7673a881b96497a22f37e3"
Nov 01 21:25:38.475906 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:38.475830    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"
Nov 01 21:25:51.628049 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:25:51.628012    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:25:52.046425 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:52.046386    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"]
Nov 01 21:25:52.098420 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:52.098388    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29ccddd-4bwgb"]
Nov 01 21:25:52.734103 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:25:52.734063    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7400cc17-91fc-4746-9c82-0a20e537c3d7" path="/var/lib/kubelet/pods/7400cc17-91fc-4746-9c82-0a20e537c3d7/volumes"
Nov 01 21:26:11.051207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:11.051166    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"]
Nov 01 21:26:11.102371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:11.102336    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05308870-bp6jp"]
Nov 01 21:26:12.729721 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:12.729686    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295c7792-fd3a-4d24-8de1-16503c246a21" path="/var/lib/kubelet/pods/295c7792-fd3a-4d24-8de1-16503c246a21/volumes"
Nov 01 21:26:14.145091 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.145062    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:26:14.145518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.145504    5066 cpu_manager.go:414] "containerMap: removing container" podUID="89525176-b0ec-468b-b745-2fb9232b9971" containerName="revoke-keys"
Nov 01 21:26:14.145570 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.145519    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89525176-b0ec-468b-b745-2fb9232b9971" containerName="revoke-keys"
Nov 01 21:26:14.145611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.145590    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="89525176-b0ec-468b-b745-2fb9232b9971" containerName="revoke-keys"
Nov 01 21:26:14.148837 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.148821    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.205788 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.205756    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-958b82-openshift-default-dockercfg-4xdj5\""
Nov 01 21:26:14.272738 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.272703    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:26:14.276646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276619    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.276769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276679    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfsq\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.276812 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276766    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.276855 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276831    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.276918 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276901    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.277019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276926    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.277019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.276965    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.277107 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.277051    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.277107 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.277101    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.379583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.379546    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.380226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.380192    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.381346 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.381325    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.381875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.381847    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.382072 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.382045    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.382212 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.382198    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.382411 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.382396    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfsq\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.382619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.382601    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.382802 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.382779    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.383010 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.381140    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.383160 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.381773    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.383638 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.383607    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.386954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.384035    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.387111 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.386452    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.411143 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.408833    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.411684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.411653    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.437362 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.437316    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.441580 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.441556    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfsq\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\") pod \"e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") " pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.461305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.461284    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:14.895806 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:26:14.895779    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0453d3_bfdc_41df_9eb1_ec28acf4ef95.slice/crio-553daaef201b97c73b89f6dae2943f3d6912d94e64fee11a6a12a0d536afa023 WatchSource:0}: Error finding container 553daaef201b97c73b89f6dae2943f3d6912d94e64fee11a6a12a0d536afa023: Status 404 returned error can't find the container with id 553daaef201b97c73b89f6dae2943f3d6912d94e64fee11a6a12a0d536afa023
Nov 01 21:26:14.896282 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.896233    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:26:14.897693 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.897668    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:14.897754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.897723    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:14.897787 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:14.897754    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:15.750486 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:15.750447    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerStarted","Data":"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"}
Nov 01 21:26:15.750901 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:15.750553    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerStarted","Data":"553daaef201b97c73b89f6dae2943f3d6912d94e64fee11a6a12a0d536afa023"}
Nov 01 21:26:15.918613 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:15.918563    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podStartSLOduration=1.918548025 podStartE2EDuration="1.918548025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:15.867241231 +0000 UTC m=+5029.252138824" watchObservedRunningTime="2026-11-01 21:26:15.918548025 +0000 UTC m=+5029.303445615"
Nov 01 21:26:16.461564 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:16.461520    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:16.466440 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:16.466412    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:16.755893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:16.755860    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:16.756947 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:16.756926    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:21.628556 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:26:21.628512    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:26:22.287773 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.287735    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:22.291705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.291684    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.356935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.351347    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:22.391109 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.391072    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.391519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.391479    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldptg\" (UniqueName: \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.394617 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.394584    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-958b82-dockercfg-d68j5\""
Nov 01 21:26:22.394748 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.394644    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-958b82\""
Nov 01 21:26:22.450954 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.450925    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:22.454631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.454608    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.494368 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.494330    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldptg\" (UniqueName: \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.494609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.494477    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.495202 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.495180    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.510189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.510157    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:22.551712 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.551629    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldptg\" (UniqueName: \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\") pod \"payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.596128 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.596091    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bp9\" (UniqueName: \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.596345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.596241    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.607640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.607597    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:22.699815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.698029    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bp9\" (UniqueName: \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.699815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.698128    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.701077 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.701019    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.756641 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.756607    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bp9\" (UniqueName: \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\") pod \"payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.767456 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.767425    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:22.924634 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.924598    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:22.985193 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:22.985156    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:22.985411 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:26:22.985383    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6776540a_b93c_4627_b00d_cd9394150245.slice/crio-53257a92bbb577ff930ae5c846bbec8dfde5f1fe52cded80c905da8be1e586f3 WatchSource:0}: Error finding container 53257a92bbb577ff930ae5c846bbec8dfde5f1fe52cded80c905da8be1e586f3: Status 404 returned error can't find the container with id 53257a92bbb577ff930ae5c846bbec8dfde5f1fe52cded80c905da8be1e586f3
Nov 01 21:26:23.030612 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.030588    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:23.142321 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.142295    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:23.142790 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:26:23.142756    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63545c51_8630_4d9b_97b6_60fd43eb79a6.slice/crio-9b4ffbdf85db52d75528257df1a609afc0eb93991876f9e206e0c532571b8fbc WatchSource:0}: Error finding container 9b4ffbdf85db52d75528257df1a609afc0eb93991876f9e206e0c532571b8fbc: Status 404 returned error can't find the container with id 9b4ffbdf85db52d75528257df1a609afc0eb93991876f9e206e0c532571b8fbc
Nov 01 21:26:23.795679 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.795639    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" event={"ID":"63545c51-8630-4d9b-97b6-60fd43eb79a6","Type":"ContainerStarted","Data":"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"}
Nov 01 21:26:23.796093 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.795703    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" event={"ID":"63545c51-8630-4d9b-97b6-60fd43eb79a6","Type":"ContainerStarted","Data":"9b4ffbdf85db52d75528257df1a609afc0eb93991876f9e206e0c532571b8fbc"}
Nov 01 21:26:23.796093 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.795741    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:23.797209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.797188    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" event={"ID":"6776540a-b93c-4627-b00d-cd9394150245","Type":"ContainerStarted","Data":"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"}
Nov 01 21:26:23.797324 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.797221    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" event={"ID":"6776540a-b93c-4627-b00d-cd9394150245","Type":"ContainerStarted","Data":"53257a92bbb577ff930ae5c846bbec8dfde5f1fe52cded80c905da8be1e586f3"}
Nov 01 21:26:23.797324 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.797250    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:23.911699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:23.911651    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" podStartSLOduration=1.9116392279999999 podStartE2EDuration="1.911639228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:23.909409475 +0000 UTC m=+5037.294307064" watchObservedRunningTime="2026-11-01 21:26:23.911639228 +0000 UTC m=+5037.296536834"
Nov 01 21:26:24.026024 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:24.025969    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" podStartSLOduration=2.025951482 podStartE2EDuration="2.025951482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:24.023056247 +0000 UTC m=+5037.407953835" watchObservedRunningTime="2026-11-01 21:26:24.025951482 +0000 UTC m=+5037.410849072"
Nov 01 21:26:28.795894 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:28.795856    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" podUID="63545c51-8630-4d9b-97b6-60fd43eb79a6" containerName="payload-processing" containerID="cri-o://396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027" gracePeriod=25
Nov 01 21:26:28.798327 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:28.798290    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" podUID="6776540a-b93c-4627-b00d-cd9394150245" containerName="payload-pre-processing" containerID="cri-o://3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef" gracePeriod=25
Nov 01 21:26:29.070192 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.070162    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:29.099893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.099867    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:29.160161 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160127    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\" (UniqueName: \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\") pod \"6776540a-b93c-4627-b00d-cd9394150245\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") "
Nov 01 21:26:29.160327 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160168    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\") pod \"63545c51-8630-4d9b-97b6-60fd43eb79a6\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") "
Nov 01 21:26:29.160327 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160270    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\") pod \"6776540a-b93c-4627-b00d-cd9394150245\" (UID: \"6776540a-b93c-4627-b00d-cd9394150245\") "
Nov 01 21:26:29.160472 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160333    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\" (UniqueName: \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\") pod \"63545c51-8630-4d9b-97b6-60fd43eb79a6\" (UID: \"63545c51-8630-4d9b-97b6-60fd43eb79a6\") "
Nov 01 21:26:29.160639 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160607    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume" pod "63545c51-8630-4d9b-97b6-60fd43eb79a6" (UID: "63545c51-8630-4d9b-97b6-60fd43eb79a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:26:29.160747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160644    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume" pod "6776540a-b93c-4627-b00d-cd9394150245" (UID: "6776540a-b93c-4627-b00d-cd9394150245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:26:29.160800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160751    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6776540a-b93c-4627-b00d-cd9394150245-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:29.160800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.160772    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63545c51-8630-4d9b-97b6-60fd43eb79a6-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:29.162350 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.162328    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg" pod "6776540a-b93c-4627-b00d-cd9394150245" (UID: "6776540a-b93c-4627-b00d-cd9394150245"). InnerVolumeSpecName "kube-api-access-ldptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:26:29.162435 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.162351    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9" pod "63545c51-8630-4d9b-97b6-60fd43eb79a6" (UID: "63545c51-8630-4d9b-97b6-60fd43eb79a6"). InnerVolumeSpecName "kube-api-access-b2bp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:26:29.261963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.261917    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bp9\" (UniqueName: \"kubernetes.io/projected/63545c51-8630-4d9b-97b6-60fd43eb79a6-kube-api-access-b2bp9\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:29.261963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.261959    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldptg\" (UniqueName: \"kubernetes.io/projected/6776540a-b93c-4627-b00d-cd9394150245-kube-api-access-ldptg\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:26:29.831596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.831565    5066 generic.go:356] "Generic (PLEG): container finished" podID="6776540a-b93c-4627-b00d-cd9394150245" containerID="3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef" exitCode=0
Nov 01 21:26:29.832046 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.831611    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"
Nov 01 21:26:29.832046 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.831643    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" event={"ID":"6776540a-b93c-4627-b00d-cd9394150245","Type":"ContainerDied","Data":"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"}
Nov 01 21:26:29.832046 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.831693    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h" event={"ID":"6776540a-b93c-4627-b00d-cd9394150245","Type":"ContainerDied","Data":"53257a92bbb577ff930ae5c846bbec8dfde5f1fe52cded80c905da8be1e586f3"}
Nov 01 21:26:29.832046 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.831716    5066 scope.go:122] "RemoveContainer" containerID="3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"
Nov 01 21:26:29.833120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.833000    5066 generic.go:356] "Generic (PLEG): container finished" podID="63545c51-8630-4d9b-97b6-60fd43eb79a6" containerID="396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027" exitCode=0
Nov 01 21:26:29.833120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.833026    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" event={"ID":"63545c51-8630-4d9b-97b6-60fd43eb79a6","Type":"ContainerDied","Data":"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"}
Nov 01 21:26:29.833120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.833041    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"
Nov 01 21:26:29.833120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.833070    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b" event={"ID":"63545c51-8630-4d9b-97b6-60fd43eb79a6","Type":"ContainerDied","Data":"9b4ffbdf85db52d75528257df1a609afc0eb93991876f9e206e0c532571b8fbc"}
Nov 01 21:26:29.842081 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.842063    5066 scope.go:122] "RemoveContainer" containerID="3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"
Nov 01 21:26:29.842351 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:26:29.842329    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef\": container with ID starting with 3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef not found: ID does not exist" containerID="3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"
Nov 01 21:26:29.842411 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.842366    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef"} err="failed to get container status \"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef\": rpc error: code = NotFound desc = could not find container \"3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef\": container with ID starting with 3365978f836e44ecf9244ee8dbd5a74a0a42fc019509ec0dc512908edbfb18ef not found: ID does not exist"
Nov 01 21:26:29.842411 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.842390    5066 scope.go:122] "RemoveContainer" containerID="396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"
Nov 01 21:26:29.850905 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.850887    5066 scope.go:122] "RemoveContainer" containerID="396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"
Nov 01 21:26:29.851159 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:26:29.851142    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027\": container with ID starting with 396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027 not found: ID does not exist" containerID="396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"
Nov 01 21:26:29.851208 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.851165    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027"} err="failed to get container status \"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027\": rpc error: code = NotFound desc = could not find container \"396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027\": container with ID starting with 396913744e9af9cead8d23415c88ac159eb60fc097af3e38ad50bb0f61ad2027 not found: ID does not exist"
Nov 01 21:26:29.998597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:29.998557    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:30.002114 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.002089    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-958b82-b95d975b5-kg67b"]
Nov 01 21:26:30.118642 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.118551    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:30.121011 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.120984    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-958b82-5f44bdf5bc-m8s4h"]
Nov 01 21:26:30.463963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.463880    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:26:30.464349 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464336    5066 cpu_manager.go:414] "containerMap: removing container" podUID="63545c51-8630-4d9b-97b6-60fd43eb79a6" containerName="payload-processing"
Nov 01 21:26:30.464388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464351    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63545c51-8630-4d9b-97b6-60fd43eb79a6" containerName="payload-processing"
Nov 01 21:26:30.464388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464366    5066 cpu_manager.go:414] "containerMap: removing container" podUID="6776540a-b93c-4627-b00d-cd9394150245" containerName="payload-pre-processing"
Nov 01 21:26:30.464388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464372    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6776540a-b93c-4627-b00d-cd9394150245" containerName="payload-pre-processing"
Nov 01 21:26:30.464516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464437    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="63545c51-8630-4d9b-97b6-60fd43eb79a6" containerName="payload-processing"
Nov 01 21:26:30.464516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.464448    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="6776540a-b93c-4627-b00d-cd9394150245" containerName="payload-pre-processing"
Nov 01 21:26:30.468937 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.468919    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.522371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.522305    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:26:30.522536 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.522380    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-958b82-serving-cert\""
Nov 01 21:26:30.522536 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.522511    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:26:30.524319 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.524294    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:26:30.570266 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.570239    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:26:30.570266 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.570258    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:26:30.575974 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.575941    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.576133 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.576113    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.576200 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.576151    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.576302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.576283    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflpp\" (UniqueName: \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.677274 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.677240    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.677430 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.677319    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.677430 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.677339    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.677430 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.677381    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflpp\" (UniqueName: \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.677934 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.677902    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.679758 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.679735    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.679856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.679790    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.730062 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.730033    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63545c51-8630-4d9b-97b6-60fd43eb79a6" path="/var/lib/kubelet/pods/63545c51-8630-4d9b-97b6-60fd43eb79a6/volumes"
Nov 01 21:26:30.730446 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.730432    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6776540a-b93c-4627-b00d-cd9394150245" path="/var/lib/kubelet/pods/6776540a-b93c-4627-b00d-cd9394150245/volumes"
Nov 01 21:26:30.734619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.734596    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflpp\" (UniqueName: \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\") pod \"maas-api-e2e-ipp-a-958b82-7784c57d48-kshct\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:30.781621 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:30.781598    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:31.163185 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:26:31.163155    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74090bba_b00b_4f54_821d_2e8017a054b5.slice/crio-c6b637588cc0df3e47453776779db58fbffe0e3bf658b6f064d7708d81cfd011 WatchSource:0}: Error finding container c6b637588cc0df3e47453776779db58fbffe0e3bf658b6f064d7708d81cfd011: Status 404 returned error can't find the container with id c6b637588cc0df3e47453776779db58fbffe0e3bf658b6f064d7708d81cfd011
Nov 01 21:26:31.163469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:31.163263    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:26:31.164723 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:31.164705    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:26:31.861871 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:31.861836    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" event={"ID":"74090bba-b00b-4f54-821d-2e8017a054b5","Type":"ContainerStarted","Data":"c6b637588cc0df3e47453776779db58fbffe0e3bf658b6f064d7708d81cfd011"}
Nov 01 21:26:32.869095 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:32.869062    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" event={"ID":"74090bba-b00b-4f54-821d-2e8017a054b5","Type":"ContainerStarted","Data":"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"}
Nov 01 21:26:32.869520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:32.869139    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:32.987131 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:32.987083    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" podStartSLOduration=2.044971975 podStartE2EDuration="2.987069559s" totalImagesPullingTime="942.097584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:32.983460694 +0000 UTC m=+5046.368358288" watchObservedRunningTime="2026-11-01 21:26:32.987069559 +0000 UTC m=+5046.371967148"
Nov 01 21:26:38.888851 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:38.888819    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:26:45.951235 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.951204    5066 generic.go:356] "Generic (PLEG): container finished" podID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerID="5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316" exitCode=0
Nov 01 21:26:45.951750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.951291    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerDied","Data":"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"}
Nov 01 21:26:45.951828 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.951764    5066 scope.go:122] "RemoveContainer" containerID="5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"
Nov 01 21:26:45.952973 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.952948    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:45.953086 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.953032    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:45.953086 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:45.953078    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:46.756484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:46.756447    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:46.959353 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:46.959314    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerStarted","Data":"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"}
Nov 01 21:26:47.476518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:47.473806    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:47.482518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:47.482049    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 01 21:26:47.482518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:47.482100    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 01 21:26:48.481938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:48.481897    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 01 21:26:48.482598 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:48.482561    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 01 21:26:49.476952 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:49.476913    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 01 21:26:49.477257 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:49.477229    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 01 21:26:50.482572 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:50.482534    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 01 21:26:50.483594 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:50.483565    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 01 21:26:51.471923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:51.471882    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 01 21:26:51.472245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:51.472213    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 01 21:26:51.627786 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:26:51.627750    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:26:52.465672 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:52.465641    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:52.466055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:52.466038    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:52.466772 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:52.466756    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:26:55.498053 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.498011    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:26:55.502144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.502114    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.555487 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.555444    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-30b7a8-openshift-default-dockercfg-9q6tc\""
Nov 01 21:26:55.572082 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572052    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572246 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572092    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6896\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572246 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572180    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572246 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572216    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572361 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572270    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572361 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572312    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572361 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572338    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572459 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572377    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.572459 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.572416    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.610089 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.610055    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:26:55.673941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.673902    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.673966    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.673994    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674132 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674016    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674147    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674184    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6896\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674229    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674256    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674297    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674323    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674394    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674790    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674812    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.674947 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.674895    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.676196 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.676178    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.676692 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.676672    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.732146 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.732114    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.732336 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.732207    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6896\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\") pod \"e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") " pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:55.827236 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:55.827158    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:56.251518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:56.251468    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:26:56.252263 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:26:56.252235    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bd1e45_47da_4bdc_b623_e4b3ac97d764.slice/crio-564a9dc4dde7ebbe4f0ebbf6bffea3c5912bfb8eede6875fb4fb66f5317f3af1 WatchSource:0}: Error finding container 564a9dc4dde7ebbe4f0ebbf6bffea3c5912bfb8eede6875fb4fb66f5317f3af1: Status 404 returned error can't find the container with id 564a9dc4dde7ebbe4f0ebbf6bffea3c5912bfb8eede6875fb4fb66f5317f3af1
Nov 01 21:26:56.254395 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:56.254368    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:56.254471 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:56.254424    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:56.254471 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:56.254454    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:26:57.045846 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:57.045808    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerStarted","Data":"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"}
Nov 01 21:26:57.046230 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:57.045864    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerStarted","Data":"564a9dc4dde7ebbe4f0ebbf6bffea3c5912bfb8eede6875fb4fb66f5317f3af1"}
Nov 01 21:26:57.214381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:57.214336    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podStartSLOduration=2.214324634 podStartE2EDuration="2.214324634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:26:57.161650613 +0000 UTC m=+5070.546548203" watchObservedRunningTime="2026-11-01 21:26:57.214324634 +0000 UTC m=+5070.599222224"
Nov 01 21:26:57.828300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:57.828269    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:57.833065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:57.833036    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:58.051411 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:58.051379    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:26:58.052342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:26:58.052324    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:27:03.459485 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.459444    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:03.462373 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.462353    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.519166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.519130    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:03.565522 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.565467    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30b7a8-dockercfg-pcz4q\""
Nov 01 21:27:03.568513 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.568087    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30b7a8\""
Nov 01 21:27:03.569220 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.569169    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl76g\" (UniqueName: \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.569406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.569348    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.620612 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.620575    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:03.623187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.623160    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.670683 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.670639    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.670883 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.670768    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl76g\" (UniqueName: \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.671583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.671554    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.673341 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.673316    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:03.728186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.728145    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl76g\" (UniqueName: \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\") pod \"payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.771575 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.771545    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8f7\" (UniqueName: \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.771771 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.771744    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.775305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.775282    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:03.873247 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.873211    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8f7\" (UniqueName: \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.873434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.873416    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.874137 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.874115    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.930747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.930710    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8f7\" (UniqueName: \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\") pod \"payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:03.935571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:03.935545    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:04.149485 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:04.149452    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:04.149721 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:27:04.149696    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fee775_78e8_4573_a213_a817c66f0d80.slice/crio-6a54b7d94e1bcca178a70d7881bf7f0d3624336be5f20c4f92fd35a9dee342d7 WatchSource:0}: Error finding container 6a54b7d94e1bcca178a70d7881bf7f0d3624336be5f20c4f92fd35a9dee342d7: Status 404 returned error can't find the container with id 6a54b7d94e1bcca178a70d7881bf7f0d3624336be5f20c4f92fd35a9dee342d7
Nov 01 21:27:04.234337 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:04.234264    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:04.305771 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:27:04.305743    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081ebb03_b2bd_4e30_99d8_695a023f41f2.slice/crio-d4a2e9a3db5b92091af311adcfbeb59bf2ba9cd49db3518cd822a326e3dcfe11 WatchSource:0}: Error finding container d4a2e9a3db5b92091af311adcfbeb59bf2ba9cd49db3518cd822a326e3dcfe11: Status 404 returned error can't find the container with id d4a2e9a3db5b92091af311adcfbeb59bf2ba9cd49db3518cd822a326e3dcfe11
Nov 01 21:27:04.306814 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:04.306788    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:04.339539 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:04.339514    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:05.091300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.091264    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" event={"ID":"081ebb03-b2bd-4e30-99d8-695a023f41f2","Type":"ContainerStarted","Data":"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"}
Nov 01 21:27:05.091878 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.091325    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" event={"ID":"081ebb03-b2bd-4e30-99d8-695a023f41f2","Type":"ContainerStarted","Data":"d4a2e9a3db5b92091af311adcfbeb59bf2ba9cd49db3518cd822a326e3dcfe11"}
Nov 01 21:27:05.091878 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.091438    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:05.093019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.092996    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" event={"ID":"56fee775-78e8-4573-a213-a817c66f0d80","Type":"ContainerStarted","Data":"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"}
Nov 01 21:27:05.093019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.093029    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" event={"ID":"56fee775-78e8-4573-a213-a817c66f0d80","Type":"ContainerStarted","Data":"6a54b7d94e1bcca178a70d7881bf7f0d3624336be5f20c4f92fd35a9dee342d7"}
Nov 01 21:27:05.093207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.093054    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:05.093207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.093146    5066 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" secret="" err="secret \"payload-processing-e2e-ipp-b-30b7a8-dockercfg-pcz4q\" not found"
Nov 01 21:27:05.188580 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:05.188539    5066 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-30b7a8: configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:05.188762 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:05.188624    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume podName:56fee775-78e8-4573-a213-a817c66f0d80 nodeName:}" failed. No retries permitted until 2026-11-01 21:27:05.688607659 +0000 UTC m=+5079.073505230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" (UID: "56fee775-78e8-4573-a213-a817c66f0d80") : configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:05.206939 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.206884    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" podStartSLOduration=2.20687065 podStartE2EDuration="2.20687065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:05.205829962 +0000 UTC m=+5078.590727552" watchObservedRunningTime="2026-11-01 21:27:05.20687065 +0000 UTC m=+5078.591768240"
Nov 01 21:27:05.321208 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:05.321149    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" podStartSLOduration=2.321134529 podStartE2EDuration="2.321134529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:05.319228235 +0000 UTC m=+5078.704125827" watchObservedRunningTime="2026-11-01 21:27:05.321134529 +0000 UTC m=+5078.706032165"
Nov 01 21:27:05.692345 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:05.692310    5066 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-30b7a8: configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:05.692512 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:05.692391    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume podName:56fee775-78e8-4573-a213-a817c66f0d80 nodeName:}" failed. No retries permitted until 2026-11-01 21:27:06.69237158 +0000 UTC m=+5080.077269165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" (UID: "56fee775-78e8-4573-a213-a817c66f0d80") : configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:06.702838 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:06.702797    5066 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-30b7a8: configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:06.703280 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:06.702874    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume podName:56fee775-78e8-4573-a213-a817c66f0d80 nodeName:}" failed. No retries permitted until 2026-11-01 21:27:08.702860182 +0000 UTC m=+5082.087757751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" (UID: "56fee775-78e8-4573-a213-a817c66f0d80") : configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:08.730042 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:08.729969    5066 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-30b7a8: configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:08.730042 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:08.730031    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume podName:56fee775-78e8-4573-a213-a817c66f0d80 nodeName:}" failed. No retries permitted until 2026-11-01 21:27:12.730016323 +0000 UTC m=+5086.114913905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" (UID: "56fee775-78e8-4573-a213-a817c66f0d80") : configmap "payload-processing-plugins-e2e-ipp-b-30b7a8" not found
Nov 01 21:27:10.091630 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.091591    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" podUID="081ebb03-b2bd-4e30-99d8-695a023f41f2" containerName="payload-processing" containerID="cri-o://dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a" gracePeriod=25
Nov 01 21:27:10.348506 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.348432    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:10.447852 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.447820    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\" (UniqueName: \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\") pod \"081ebb03-b2bd-4e30-99d8-695a023f41f2\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") "
Nov 01 21:27:10.447852 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.447853    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\") pod \"081ebb03-b2bd-4e30-99d8-695a023f41f2\" (UID: \"081ebb03-b2bd-4e30-99d8-695a023f41f2\") "
Nov 01 21:27:10.448249 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.448227    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume" pod "081ebb03-b2bd-4e30-99d8-695a023f41f2" (UID: "081ebb03-b2bd-4e30-99d8-695a023f41f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:27:10.449856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.449837    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7" pod "081ebb03-b2bd-4e30-99d8-695a023f41f2" (UID: "081ebb03-b2bd-4e30-99d8-695a023f41f2"). InnerVolumeSpecName "kube-api-access-lr8f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:27:10.548981 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.548946    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr8f7\" (UniqueName: \"kubernetes.io/projected/081ebb03-b2bd-4e30-99d8-695a023f41f2-kube-api-access-lr8f7\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:10.548981 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:10.548980    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081ebb03-b2bd-4e30-99d8-695a023f41f2-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:11.100039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.099925    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" podUID="56fee775-78e8-4573-a213-a817c66f0d80" containerName="payload-pre-processing" containerID="cri-o://1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc" gracePeriod=25
Nov 01 21:27:11.128771 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.128744    5066 generic.go:356] "Generic (PLEG): container finished" podID="081ebb03-b2bd-4e30-99d8-695a023f41f2" containerID="dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a" exitCode=0
Nov 01 21:27:11.128887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.128787    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"
Nov 01 21:27:11.128887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.128827    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" event={"ID":"081ebb03-b2bd-4e30-99d8-695a023f41f2","Type":"ContainerDied","Data":"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"}
Nov 01 21:27:11.128887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.128877    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h" event={"ID":"081ebb03-b2bd-4e30-99d8-695a023f41f2","Type":"ContainerDied","Data":"d4a2e9a3db5b92091af311adcfbeb59bf2ba9cd49db3518cd822a326e3dcfe11"}
Nov 01 21:27:11.129089 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.128899    5066 scope.go:122] "RemoveContainer" containerID="dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"
Nov 01 21:27:11.212183 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.212157    5066 scope.go:122] "RemoveContainer" containerID="dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"
Nov 01 21:27:11.212474 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:11.212450    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a\": container with ID starting with dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a not found: ID does not exist" containerID="dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"
Nov 01 21:27:11.212600 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.212482    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a"} err="failed to get container status \"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a\": rpc error: code = NotFound desc = could not find container \"dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a\": container with ID starting with dc84fe3bf4ec997bab57629ca57646c50bdff0e6af886d83332e992e4edbf91a not found: ID does not exist"
Nov 01 21:27:11.296240 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.296209    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:11.346558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.346528    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30b7a8-6bbdddcbf7-cqm8h"]
Nov 01 21:27:11.368259 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.368234    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:11.404970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.404939    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:27:11.405385 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405372    5066 cpu_manager.go:414] "containerMap: removing container" podUID="081ebb03-b2bd-4e30-99d8-695a023f41f2" containerName="payload-processing"
Nov 01 21:27:11.405441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405386    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081ebb03-b2bd-4e30-99d8-695a023f41f2" containerName="payload-processing"
Nov 01 21:27:11.405441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405396    5066 cpu_manager.go:414] "containerMap: removing container" podUID="56fee775-78e8-4573-a213-a817c66f0d80" containerName="payload-pre-processing"
Nov 01 21:27:11.405441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405401    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fee775-78e8-4573-a213-a817c66f0d80" containerName="payload-pre-processing"
Nov 01 21:27:11.405593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405504    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="081ebb03-b2bd-4e30-99d8-695a023f41f2" containerName="payload-processing"
Nov 01 21:27:11.405593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.405513    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fee775-78e8-4573-a213-a817c66f0d80" containerName="payload-pre-processing"
Nov 01 21:27:11.408563 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.408544    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.458378 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.458353    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\") pod \"56fee775-78e8-4573-a213-a817c66f0d80\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") "
Nov 01 21:27:11.458548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.458419    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\" (UniqueName: \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\") pod \"56fee775-78e8-4573-a213-a817c66f0d80\" (UID: \"56fee775-78e8-4573-a213-a817c66f0d80\") "
Nov 01 21:27:11.458858 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.458815    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume" pod "56fee775-78e8-4573-a213-a817c66f0d80" (UID: "56fee775-78e8-4573-a213-a817c66f0d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:27:11.461328 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.461297    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g" pod "56fee775-78e8-4573-a213-a817c66f0d80" (UID: "56fee775-78e8-4573-a213-a817c66f0d80"). InnerVolumeSpecName "kube-api-access-pl76g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:27:11.461687 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.461664    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-30b7a8-serving-cert\""
Nov 01 21:27:11.465904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.465873    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:27:11.559489 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559455    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbp2q\" (UniqueName: \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.559668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559513    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.559668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559614    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.559848 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559724    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.559914 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559888    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl76g\" (UniqueName: \"kubernetes.io/projected/56fee775-78e8-4573-a213-a817c66f0d80-kube-api-access-pl76g\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:11.559914 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.559909    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fee775-78e8-4573-a213-a817c66f0d80-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:27:11.661013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.660897    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp2q\" (UniqueName: \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.661013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.660958    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.661013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.660990    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.661522 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.661127    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.661522 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:11.661383    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-serving-cert: secret "maas-api-e2e-ipp-b-30b7a8-serving-cert" not found
Nov 01 21:27:11.661522 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:11.661447    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls podName:31e94302-e5bd-4a9d-a135-23c0a55f4d2b nodeName:}" failed. No retries permitted until 2026-11-01 21:27:12.16142671 +0000 UTC m=+5085.546324291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls") pod "maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" (UID: "31e94302-e5bd-4a9d-a135-23c0a55f4d2b") : secret "maas-api-e2e-ipp-b-30b7a8-serving-cert" not found
Nov 01 21:27:11.662622 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.662596    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.666188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.666150    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:11.721178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:11.721152    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbp2q\" (UniqueName: \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:12.135991 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.135954    5066 generic.go:356] "Generic (PLEG): container finished" podID="56fee775-78e8-4573-a213-a817c66f0d80" containerID="1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc" exitCode=0
Nov 01 21:27:12.136374 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.136004    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"
Nov 01 21:27:12.136374 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.136040    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" event={"ID":"56fee775-78e8-4573-a213-a817c66f0d80","Type":"ContainerDied","Data":"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"}
Nov 01 21:27:12.136374 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.136091    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd" event={"ID":"56fee775-78e8-4573-a213-a817c66f0d80","Type":"ContainerDied","Data":"6a54b7d94e1bcca178a70d7881bf7f0d3624336be5f20c4f92fd35a9dee342d7"}
Nov 01 21:27:12.136374 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.136119    5066 scope.go:122] "RemoveContainer" containerID="1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"
Nov 01 21:27:12.145994 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.145972    5066 scope.go:122] "RemoveContainer" containerID="1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"
Nov 01 21:27:12.146253 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:12.146234    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc\": container with ID starting with 1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc not found: ID does not exist" containerID="1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"
Nov 01 21:27:12.146307 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.146261    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc"} err="failed to get container status \"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc\": rpc error: code = NotFound desc = could not find container \"1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc\": container with ID starting with 1d4a6f11cace92c424f4e2930c431fe8016be5a457cbd547dcff4830b5c799bc not found: ID does not exist"
Nov 01 21:27:12.165947 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.165925    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:12.168341 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.168320    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:12.302423 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.302392    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:12.307642 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.307617    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30b7a8-69bb498cf5-rslrd"]
Nov 01 21:27:12.321264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.321229    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:12.730424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.730387    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081ebb03-b2bd-4e30-99d8-695a023f41f2" path="/var/lib/kubelet/pods/081ebb03-b2bd-4e30-99d8-695a023f41f2/volumes"
Nov 01 21:27:12.730809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.730795    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fee775-78e8-4573-a213-a817c66f0d80" path="/var/lib/kubelet/pods/56fee775-78e8-4573-a213-a817c66f0d80/volumes"
Nov 01 21:27:12.746874 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:27:12.746846    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e94302_e5bd_4a9d_a135_23c0a55f4d2b.slice/crio-20bef046b3b1a60bcaf9277929cdd2e1f3e409883ecb4c0924ac97c0866fc6bc WatchSource:0}: Error finding container 20bef046b3b1a60bcaf9277929cdd2e1f3e409883ecb4c0924ac97c0866fc6bc: Status 404 returned error can't find the container with id 20bef046b3b1a60bcaf9277929cdd2e1f3e409883ecb4c0924ac97c0866fc6bc
Nov 01 21:27:12.747009 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:12.746899    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:27:13.143551 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:13.143436    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" event={"ID":"31e94302-e5bd-4a9d-a135-23c0a55f4d2b","Type":"ContainerStarted","Data":"20bef046b3b1a60bcaf9277929cdd2e1f3e409883ecb4c0924ac97c0866fc6bc"}
Nov 01 21:27:14.151292 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:14.151256    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" event={"ID":"31e94302-e5bd-4a9d-a135-23c0a55f4d2b","Type":"ContainerStarted","Data":"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"}
Nov 01 21:27:14.151886 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:14.151326    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:14.269454 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:14.269403    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" podStartSLOduration=2.501091798 podStartE2EDuration="3.269390328s" totalImagesPullingTime="768.29853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:14.265899993 +0000 UTC m=+5087.650797585" watchObservedRunningTime="2026-11-01 21:27:14.269390328 +0000 UTC m=+5087.654287918"
Nov 01 21:27:20.182682 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:20.182639    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:27:21.629451 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:21.629406    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:27:27.520938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:27.520908    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:27:27.578721 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:27.578660    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:27:28.052745 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.052710    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:28.052927 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.052774    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:28.245870 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.245840    5066 generic.go:356] "Generic (PLEG): container finished" podID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerID="de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c" exitCode=0
Nov 01 21:27:28.246027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.245887    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerDied","Data":"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"}
Nov 01 21:27:28.246277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.246263    5066 scope.go:122] "RemoveContainer" containerID="de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"
Nov 01 21:27:28.247513 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.247466    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:27:28.247583 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.247556    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:27:28.247619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.247589    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:27:28.269057 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.269033    5066 scope.go:122] "RemoveContainer" containerID="1c01337619ce619ab368b692264d20b6f125a6a6c7eb7deab3a2996bee4c34d0"
Nov 01 21:27:28.284193 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:28.284169    5066 scope.go:122] "RemoveContainer" containerID="21722fbd1fe9fb7b8666a804d7affad1294d1460183edfd40e501bffe7b93f88"
Nov 01 21:27:29.252889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:29.252846    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerStarted","Data":"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"}
Nov 01 21:27:29.827753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:29.827718    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:27:29.827933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:29.827888    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:29.827977 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:29.827941    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:30.827962 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:30.827925    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:30.828465 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:30.828000    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:31.827828 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:31.827790    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:31.828000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:31.827840    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:32.827798 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:32.827762    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:32.827996 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:32.827823    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:33.828078 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:33.828044    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Nov 01 21:27:33.828539 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:33.828098    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Nov 01 21:27:34.831285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:34.831251    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:27:34.831769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:34.831720    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:27:34.832372 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:34.832353    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:27:47.043101 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:47.043075    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fhnrf_c758dba3-c56f-4436-af91-e25f42d98603/payload-processing/0.log"
Nov 01 21:27:47.259406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:27:47.259380    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fhnrf_c758dba3-c56f-4436-af91-e25f42d98603/payload-processing/0.log"
Nov 01 21:27:51.628151 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:27:51.628109    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:28:21.628256 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:28:21.628216    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:28:48.400867 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:28:48.400798    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fhnrf_c758dba3-c56f-4436-af91-e25f42d98603/payload-processing/0.log"
Nov 01 21:28:48.604542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:28:48.604509    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fhnrf_c758dba3-c56f-4436-af91-e25f42d98603/payload-processing/0.log"
Nov 01 21:28:51.628817 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:28:51.628776    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:29:21.629482 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:29:21.629440    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:29:43.577139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:43.577102    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:29:43.577705 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:43.577326    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" containerName="maas-api" containerID="cri-o://039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938" gracePeriod=30
Nov 01 21:29:43.875626 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:43.875529    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.180:8443/health\": dial tcp 10.132.0.180:8443: connect: connection refused"
Nov 01 21:29:44.819632 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.819605    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:29:44.930650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.930558    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\" (UniqueName: \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\") pod \"74090bba-b00b-4f54-821d-2e8017a054b5\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") "
Nov 01 21:29:44.930804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.930671    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\") pod \"74090bba-b00b-4f54-821d-2e8017a054b5\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") "
Nov 01 21:29:44.930804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.930707    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\") pod \"74090bba-b00b-4f54-821d-2e8017a054b5\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") "
Nov 01 21:29:44.930804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.930789    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\") pod \"74090bba-b00b-4f54-821d-2e8017a054b5\" (UID: \"74090bba-b00b-4f54-821d-2e8017a054b5\") "
Nov 01 21:29:44.931071 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.931042    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca" pod "74090bba-b00b-4f54-821d-2e8017a054b5" (UID: "74090bba-b00b-4f54-821d-2e8017a054b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:29:44.931304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.931280    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74090bba-b00b-4f54-821d-2e8017a054b5-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:44.932897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.932860    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls" pod "74090bba-b00b-4f54-821d-2e8017a054b5" (UID: "74090bba-b00b-4f54-821d-2e8017a054b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:29:44.933008 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.932951    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs" pod "74090bba-b00b-4f54-821d-2e8017a054b5" (UID: "74090bba-b00b-4f54-821d-2e8017a054b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:29:44.933265 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:44.933243    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp" pod "74090bba-b00b-4f54-821d-2e8017a054b5" (UID: "74090bba-b00b-4f54-821d-2e8017a054b5"). InnerVolumeSpecName "kube-api-access-lflpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:29:45.032844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.032809    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflpp\" (UniqueName: \"kubernetes.io/projected/74090bba-b00b-4f54-821d-2e8017a054b5-kube-api-access-lflpp\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:45.032844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.032840    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:45.032844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.032851    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74090bba-b00b-4f54-821d-2e8017a054b5-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:45.054542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.054474    5066 generic.go:356] "Generic (PLEG): container finished" podID="74090bba-b00b-4f54-821d-2e8017a054b5" containerID="039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938" exitCode=0
Nov 01 21:29:45.054542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.054525    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" event={"ID":"74090bba-b00b-4f54-821d-2e8017a054b5","Type":"ContainerDied","Data":"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"}
Nov 01 21:29:45.054809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.054543    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"
Nov 01 21:29:45.054809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.054589    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct" event={"ID":"74090bba-b00b-4f54-821d-2e8017a054b5","Type":"ContainerDied","Data":"c6b637588cc0df3e47453776779db58fbffe0e3bf658b6f064d7708d81cfd011"}
Nov 01 21:29:45.054809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.054618    5066 scope.go:122] "RemoveContainer" containerID="039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"
Nov 01 21:29:45.064935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.064917    5066 scope.go:122] "RemoveContainer" containerID="039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"
Nov 01 21:29:45.065225 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:29:45.065205    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938\": container with ID starting with 039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938 not found: ID does not exist" containerID="039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"
Nov 01 21:29:45.065277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.065233    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938"} err="failed to get container status \"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938\": rpc error: code = NotFound desc = could not find container \"039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938\": container with ID starting with 039809334d7e3a03c85a1755f047f97b5f8ee7bfae5011867148bf29b4011938 not found: ID does not exist"
Nov 01 21:29:45.221371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.221333    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:29:45.226263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:45.226234    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-958b82-7784c57d48-kshct"]
Nov 01 21:29:46.730361 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:46.730327    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" path="/var/lib/kubelet/pods/74090bba-b00b-4f54-821d-2e8017a054b5/volumes"
Nov 01 21:29:50.760646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:50.760607    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:29:50.761103 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:50.760869    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" containerID="cri-o://4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45" gracePeriod=30
Nov 01 21:29:51.628980 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:29:51.628942    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:29:52.469328 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:52.469294    5066 patch_prober.go:29] interesting pod/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:29:52.469762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:52.469350    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:29:56.021307 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.021276    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:29:56.129461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.129379    5066 generic.go:356] "Generic (PLEG): container finished" podID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerID="4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45" exitCode=0
Nov 01 21:29:56.129461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.129430    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"
Nov 01 21:29:56.129461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.129451    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerDied","Data":"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"}
Nov 01 21:29:56.129724 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.129489    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw" event={"ID":"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95","Type":"ContainerDied","Data":"553daaef201b97c73b89f6dae2943f3d6912d94e64fee11a6a12a0d536afa023"}
Nov 01 21:29:56.129724 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.129518    5066 scope.go:122] "RemoveContainer" containerID="4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"
Nov 01 21:29:56.138769 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.138747    5066 scope.go:122] "RemoveContainer" containerID="5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"
Nov 01 21:29:56.147603 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.147579    5066 scope.go:122] "RemoveContainer" containerID="4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"
Nov 01 21:29:56.147837 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.147814    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.147935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.147856    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.147935 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.147891    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148054 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.147978    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148054 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148022    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148050    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148167 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:29:56.148050    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45\": container with ID starting with 4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45 not found: ID does not exist" containerID="4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"
Nov 01 21:29:56.148167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148086    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45"} err="failed to get container status \"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45\": rpc error: code = NotFound desc = could not find container \"4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45\": container with ID starting with 4f56a62e940292c5122499de3c6c28257a4faa801f6de4dd422396b65084bf45 not found: ID does not exist"
Nov 01 21:29:56.148167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148105    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148167 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148110    5066 scope.go:122] "RemoveContainer" containerID="5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"
Nov 01 21:29:56.148448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148167    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148247    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\") pod \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\" (UID: \"5c0453d3-bfdc-41df-9eb1-ec28acf4ef95\") "
Nov 01 21:29:56.148448 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148328    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:29:56.148754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148731    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.148903 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148840    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:29:56.148903 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148893    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:29:56.149031 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148932    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:29:56.149031 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.148999    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:29:56.149142 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:29:56.149034    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316\": container with ID starting with 5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316 not found: ID does not exist" containerID="5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"
Nov 01 21:29:56.149142 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.149065    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316"} err="failed to get container status \"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316\": rpc error: code = NotFound desc = could not find container \"5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316\": container with ID starting with 5a86e57b3672149f7c969d15e07b37c8224907e4982432562f56ebbbd28b5316 not found: ID does not exist"
Nov 01 21:29:56.150227 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.150176    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "kube-api-access-lbfsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:29:56.150596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.150568    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:29:56.150743 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.150723    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:29:56.150883 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.150855    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token" pod "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" (UID: "5c0453d3-bfdc-41df-9eb1-ec28acf4ef95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:29:56.250264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250219    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfsq\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-kube-api-access-lbfsq\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250257    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250267    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250264 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250276    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250285    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250293    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250302    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.250577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.250310    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:29:56.554955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.554915    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:29:56.558455 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.558429    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-958b82-openshift-default-75f6bc7c75-rl4dw"]
Nov 01 21:29:56.730577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:29:56.730542    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" path="/var/lib/kubelet/pods/5c0453d3-bfdc-41df-9eb1-ec28acf4ef95/volumes"
Nov 01 21:30:07.747305 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:07.747268    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:30:07.747832 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:07.747474    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" podUID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" containerName="maas-api" containerID="cri-o://cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130" gracePeriod=30
Nov 01 21:30:08.009728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.009662    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:30:08.176927 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.176894    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\") pod \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") "
Nov 01 21:30:08.177105 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.177042    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\") pod \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") "
Nov 01 21:30:08.177105 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.177071    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\" (UniqueName: \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\") pod \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") "
Nov 01 21:30:08.177213 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.177126    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") pod \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\" (UID: \"31e94302-e5bd-4a9d-a135-23c0a55f4d2b\") "
Nov 01 21:30:08.177464 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.177436    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca" pod "31e94302-e5bd-4a9d-a135-23c0a55f4d2b" (UID: "31e94302-e5bd-4a9d-a135-23c0a55f4d2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:30:08.177639 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.177538    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:08.179109 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.179082    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs" pod "31e94302-e5bd-4a9d-a135-23c0a55f4d2b" (UID: "31e94302-e5bd-4a9d-a135-23c0a55f4d2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:30:08.179209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.179177    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q" pod "31e94302-e5bd-4a9d-a135-23c0a55f4d2b" (UID: "31e94302-e5bd-4a9d-a135-23c0a55f4d2b"). InnerVolumeSpecName "kube-api-access-fbp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:30:08.179209 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.179180    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls" pod "31e94302-e5bd-4a9d-a135-23c0a55f4d2b" (UID: "31e94302-e5bd-4a9d-a135-23c0a55f4d2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:30:08.200094 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.200063    5066 generic.go:356] "Generic (PLEG): container finished" podID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" containerID="cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130" exitCode=0
Nov 01 21:30:08.200256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.200111    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"
Nov 01 21:30:08.200256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.200144    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" event={"ID":"31e94302-e5bd-4a9d-a135-23c0a55f4d2b","Type":"ContainerDied","Data":"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"}
Nov 01 21:30:08.200256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.200210    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf" event={"ID":"31e94302-e5bd-4a9d-a135-23c0a55f4d2b","Type":"ContainerDied","Data":"20bef046b3b1a60bcaf9277929cdd2e1f3e409883ecb4c0924ac97c0866fc6bc"}
Nov 01 21:30:08.200436 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.200240    5066 scope.go:122] "RemoveContainer" containerID="cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"
Nov 01 21:30:08.210869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.210844    5066 scope.go:122] "RemoveContainer" containerID="cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"
Nov 01 21:30:08.211162 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:08.211143    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130\": container with ID starting with cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130 not found: ID does not exist" containerID="cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"
Nov 01 21:30:08.211272 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.211172    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130"} err="failed to get container status \"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130\": rpc error: code = NotFound desc = could not find container \"cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130\": container with ID starting with cd6bb0d4fce0a632b6856c997846389e7f62ad476c6e7fe52a1d7a3eb5b17130 not found: ID does not exist"
Nov 01 21:30:08.279070 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.278984    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbp2q\" (UniqueName: \"kubernetes.io/projected/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-kube-api-access-fbp2q\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:08.279070 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.279023    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:08.279070 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.279038    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e94302-e5bd-4a9d-a135-23c0a55f4d2b-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:08.366781 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.366750    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:30:08.370271 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.370242    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30b7a8-6f6d448444-q5llf"]
Nov 01 21:30:08.731928 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:08.731889    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" path="/var/lib/kubelet/pods/31e94302-e5bd-4a9d-a135-23c0a55f4d2b/volumes"
Nov 01 21:30:13.045103 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:13.045067    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"]
Nov 01 21:30:13.095957 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:13.095918    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xwvr7"]
Nov 01 21:30:14.730544 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:14.730510    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77d2c20-ceb6-43b1-a65e-912e93f74e3c" path="/var/lib/kubelet/pods/c77d2c20-ceb6-43b1-a65e-912e93f74e3c/volumes"
Nov 01 21:30:17.718872 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:17.718835    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:30:17.719339 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:17.719034    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" containerID="cri-o://8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830" gracePeriod=30
Nov 01 21:30:19.496715 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.496681    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:30:19.497139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497126    5066 cpu_manager.go:414] "containerMap: removing container" podUID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" containerName="maas-api"
Nov 01 21:30:19.497186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497140    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" containerName="maas-api"
Nov 01 21:30:19.497186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497151    5066 cpu_manager.go:414] "containerMap: removing container" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:19.497186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497156    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:19.497186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497169    5066 cpu_manager.go:414] "containerMap: removing container" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:19.497186 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497174    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:19.497345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497193    5066 cpu_manager.go:414] "containerMap: removing container" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" containerName="maas-api"
Nov 01 21:30:19.497345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497198    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" containerName="maas-api"
Nov 01 21:30:19.497345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497262    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="74090bba-b00b-4f54-821d-2e8017a054b5" containerName="maas-api"
Nov 01 21:30:19.497345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497274    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e94302-e5bd-4a9d-a135-23c0a55f4d2b" containerName="maas-api"
Nov 01 21:30:19.497345 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.497279    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:19.500718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.500703    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.555544 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.555509    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:30:19.616096 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.616060    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.616300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.616198    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkgw\" (UniqueName: \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.717704 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.717666    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.717880 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.717755    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkgw\" (UniqueName: \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.718410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.718388    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.775124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.775051    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkgw\" (UniqueName: \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\") pod \"payload-pre-processing-54c4dc45d9-zw2xh\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") " pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.813253 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.813213    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:19.834171 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.834139    5066 patch_prober.go:29] interesting pod/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:30:19.834298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:19.834194    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:30:20.187679 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:30:20.187639    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec55dba_a534_4d06_9ab8_071724ec0641.slice/crio-cb9438fa43603c4415d33ab363a77de4bd2e73a243df49852f74473fc658d1aa WatchSource:0}: Error finding container cb9438fa43603c4415d33ab363a77de4bd2e73a243df49852f74473fc658d1aa: Status 404 returned error can't find the container with id cb9438fa43603c4415d33ab363a77de4bd2e73a243df49852f74473fc658d1aa
Nov 01 21:30:20.189028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:20.188995    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:30:20.270568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:20.270530    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" event={"ID":"1ec55dba-a534-4d06-9ab8-071724ec0641","Type":"ContainerStarted","Data":"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"}
Nov 01 21:30:20.270761 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:20.270598    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" event={"ID":"1ec55dba-a534-4d06-9ab8-071724ec0641","Type":"ContainerStarted","Data":"cb9438fa43603c4415d33ab363a77de4bd2e73a243df49852f74473fc658d1aa"}
Nov 01 21:30:20.270761 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:20.270696    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:20.385595 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:20.385544    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" podStartSLOduration=1.385530623 podStartE2EDuration="1.385530623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:20.38351997 +0000 UTC m=+5273.768417563" watchObservedRunningTime="2026-11-01 21:30:20.385530623 +0000 UTC m=+5273.770428212"
Nov 01 21:30:21.629362 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:21.629326    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:30:22.979321 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:22.979289    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:30:23.049360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049277    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049336    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049359    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049427    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049466    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049487    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049537    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049571    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.049700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049637    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\") pod \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\" (UID: \"c2bd1e45-47da-4bdc-b623-e4b3ac97d764\") "
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049823    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049915    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.049954    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.050024    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.050061    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.050068    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.050088    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.050120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.050101    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.051802 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.051778    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:30:23.051931 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.051913    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "kube-api-access-b6896". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:30:23.052026 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.052007    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:30:23.052121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.052101    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token" pod "c2bd1e45-47da-4bdc-b623-e4b3ac97d764" (UID: "c2bd1e45-47da-4bdc-b623-e4b3ac97d764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:30:23.150784 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150747    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6896\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-kube-api-access-b6896\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.150784 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150784    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.151004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150794    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.151004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150803    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.151004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150812    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.151004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.150821    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2bd1e45-47da-4bdc-b623-e4b3ac97d764-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:23.290266 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.290229    5066 generic.go:356] "Generic (PLEG): container finished" podID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerID="8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830" exitCode=0
Nov 01 21:30:23.290266 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.290273    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"
Nov 01 21:30:23.290488 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.290312    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerDied","Data":"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"}
Nov 01 21:30:23.290488 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.290368    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h" event={"ID":"c2bd1e45-47da-4bdc-b623-e4b3ac97d764","Type":"ContainerDied","Data":"564a9dc4dde7ebbe4f0ebbf6bffea3c5912bfb8eede6875fb4fb66f5317f3af1"}
Nov 01 21:30:23.290488 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.290390    5066 scope.go:122] "RemoveContainer" containerID="8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"
Nov 01 21:30:23.300652 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.300634    5066 scope.go:122] "RemoveContainer" containerID="de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"
Nov 01 21:30:23.309187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.309169    5066 scope.go:122] "RemoveContainer" containerID="8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"
Nov 01 21:30:23.309470 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:23.309450    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830\": container with ID starting with 8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830 not found: ID does not exist" containerID="8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"
Nov 01 21:30:23.309591 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.309479    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830"} err="failed to get container status \"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830\": rpc error: code = NotFound desc = could not find container \"8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830\": container with ID starting with 8baefb66eacc661e19aa827e04d2ff253b3706f0fc8434af284207dd4120a830 not found: ID does not exist"
Nov 01 21:30:23.309637 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.309595    5066 scope.go:122] "RemoveContainer" containerID="de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"
Nov 01 21:30:23.309836 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:23.309815    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c\": container with ID starting with de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c not found: ID does not exist" containerID="de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"
Nov 01 21:30:23.309904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.309846    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c"} err="failed to get container status \"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c\": rpc error: code = NotFound desc = could not find container \"de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c\": container with ID starting with de4f5716f532415028b19bdd05183a6c8dbad7c15a7f606f557749231eaa068c not found: ID does not exist"
Nov 01 21:30:23.460704 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.460668    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:30:23.464019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:23.463998    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-30b7a8-openshift-default-6bb47cf4f9-5q82h"]
Nov 01 21:30:24.730304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:24.730268    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" path="/var/lib/kubelet/pods/c2bd1e45-47da-4bdc-b623-e4b3ac97d764/volumes"
Nov 01 21:30:31.279269 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:31.279238    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:30:33.378437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378399    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:30:33.378864 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378847    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.378864 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378858    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.378938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378866    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.378938 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378872    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.379004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378968    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.379004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378976    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bd1e45-47da-4bdc-b623-e4b3ac97d764" containerName="istio-proxy"
Nov 01 21:30:33.379004 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.378987    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0453d3-bfdc-41df-9eb1-ec28acf4ef95" containerName="istio-proxy"
Nov 01 21:30:33.383715 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.383693    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.437589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.437549    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:30:33.451810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.451777    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.451972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.451846    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lxl\" (UniqueName: \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.553000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.552961    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.553195 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.553013    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lxl\" (UniqueName: \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.553711 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.553689    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.610796 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.610768    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lxl\" (UniqueName: \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\") pod \"payload-processing-56645ccfc8-m6sg4\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") " pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:33.696483 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:33.696386    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:34.069719 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:34.069686    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:30:34.070563 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:30:34.070529    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60c20a3_7577_4dc8_83a7_b7668d8bac05.slice/crio-6756f22742d3651aa1f79f39de05e0d2b6e4a3d70bfa688f84943b8558dbdd90 WatchSource:0}: Error finding container 6756f22742d3651aa1f79f39de05e0d2b6e4a3d70bfa688f84943b8558dbdd90: Status 404 returned error can't find the container with id 6756f22742d3651aa1f79f39de05e0d2b6e4a3d70bfa688f84943b8558dbdd90
Nov 01 21:30:34.358713 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:34.358617    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" event={"ID":"e60c20a3-7577-4dc8-83a7-b7668d8bac05","Type":"ContainerStarted","Data":"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"}
Nov 01 21:30:34.358713 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:34.358685    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" event={"ID":"e60c20a3-7577-4dc8-83a7-b7668d8bac05","Type":"ContainerStarted","Data":"6756f22742d3651aa1f79f39de05e0d2b6e4a3d70bfa688f84943b8558dbdd90"}
Nov 01 21:30:34.358939 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:34.358820    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:34.477368 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:34.477312    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" podStartSLOduration=1.477296333 podStartE2EDuration="1.477296333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:34.474251901 +0000 UTC m=+5287.859149485" watchObservedRunningTime="2026-11-01 21:30:34.477296333 +0000 UTC m=+5287.862193922"
Nov 01 21:30:39.048152 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:39.048110    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"]
Nov 01 21:30:39.096609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:39.096577    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cmf8z"]
Nov 01 21:30:40.729801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:40.729762    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89525176-b0ec-468b-b745-2fb9232b9971" path="/var/lib/kubelet/pods/89525176-b0ec-468b-b745-2fb9232b9971/volumes"
Nov 01 21:30:45.367551 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:45.367519    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:30:45.539180 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:45.539139    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 21:30:47.606558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.606516    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7f5ccd4859-xftxz"]
Nov 01 21:30:47.612238 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.612212    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.662083 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.662051    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7f5ccd4859-xftxz"]
Nov 01 21:30:47.761336 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.761302    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7d98d48688-6pcws"]
Nov 01 21:30:47.765002 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.764982    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:47.796073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.796039    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvv8r\" (UniqueName: \"kubernetes.io/projected/939548ca-a52e-46d5-ae33-f57e79fefd64-kube-api-access-jvv8r\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.796250 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.796159    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939548ca-a52e-46d5-ae33-f57e79fefd64-plugins-config-volume\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.821753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.821718    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7d98d48688-6pcws"]
Nov 01 21:30:47.897574 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.897460    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvv8r\" (UniqueName: \"kubernetes.io/projected/939548ca-a52e-46d5-ae33-f57e79fefd64-kube-api-access-jvv8r\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.897743 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.897637    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs59t\" (UniqueName: \"kubernetes.io/projected/0164ec25-f653-42a1-9860-8470c532ee23-kube-api-access-cs59t\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:47.897808 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.897786    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0164ec25-f653-42a1-9860-8470c532ee23-plugins-config-volume\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:47.897882 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.897865    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939548ca-a52e-46d5-ae33-f57e79fefd64-plugins-config-volume\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.898540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.898485    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939548ca-a52e-46d5-ae33-f57e79fefd64-plugins-config-volume\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.955573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.955533    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvv8r\" (UniqueName: \"kubernetes.io/projected/939548ca-a52e-46d5-ae33-f57e79fefd64-kube-api-access-jvv8r\") pod \"payload-processing-7f5ccd4859-xftxz\" (UID: \"939548ca-a52e-46d5-ae33-f57e79fefd64\") " pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:47.999344 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.999304    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs59t\" (UniqueName: \"kubernetes.io/projected/0164ec25-f653-42a1-9860-8470c532ee23-kube-api-access-cs59t\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:47.999576 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:47.999357    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0164ec25-f653-42a1-9860-8470c532ee23-plugins-config-volume\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:48.000101 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.000078    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0164ec25-f653-42a1-9860-8470c532ee23-plugins-config-volume\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:48.057429 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.057398    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs59t\" (UniqueName: \"kubernetes.io/projected/0164ec25-f653-42a1-9860-8470c532ee23-kube-api-access-cs59t\") pod \"payload-pre-processing-7d98d48688-6pcws\" (UID: \"0164ec25-f653-42a1-9860-8470c532ee23\") " pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:48.078256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.078221    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:48.227474 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.227445    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:48.455351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.455320    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7d98d48688-6pcws"]
Nov 01 21:30:48.455615 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:30:48.455593    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0164ec25_f653_42a1_9860_8470c532ee23.slice/crio-a2a146267fe3a32eacacdc7ecd555f908d4a51d7eb8144f33e95b2d03ab7b666 WatchSource:0}: Error finding container a2a146267fe3a32eacacdc7ecd555f908d4a51d7eb8144f33e95b2d03ab7b666: Status 404 returned error can't find the container with id a2a146267fe3a32eacacdc7ecd555f908d4a51d7eb8144f33e95b2d03ab7b666
Nov 01 21:30:48.600277 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:48.600244    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7f5ccd4859-xftxz"]
Nov 01 21:30:48.600648 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:30:48.600626    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939548ca_a52e_46d5_ae33_f57e79fefd64.slice/crio-b3e41437c552f9afe813941c647ff5a8cc44119921a0ada3a1b8c786cfa8ffe0 WatchSource:0}: Error finding container b3e41437c552f9afe813941c647ff5a8cc44119921a0ada3a1b8c786cfa8ffe0: Status 404 returned error can't find the container with id b3e41437c552f9afe813941c647ff5a8cc44119921a0ada3a1b8c786cfa8ffe0
Nov 01 21:30:49.441297 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.441260    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz" event={"ID":"939548ca-a52e-46d5-ae33-f57e79fefd64","Type":"ContainerStarted","Data":"30426266d2140c743446424c22460d030fc14e5ec79404880cf75936739ae575"}
Nov 01 21:30:49.441811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.441313    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz" event={"ID":"939548ca-a52e-46d5-ae33-f57e79fefd64","Type":"ContainerStarted","Data":"b3e41437c552f9afe813941c647ff5a8cc44119921a0ada3a1b8c786cfa8ffe0"}
Nov 01 21:30:49.441811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.441374    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:30:49.442934 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.442914    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws" event={"ID":"0164ec25-f653-42a1-9860-8470c532ee23","Type":"ContainerStarted","Data":"cef231d69e21cf48af3564bd21b7686b137e66b534e8bfdf12dbdb152eda66aa"}
Nov 01 21:30:49.443055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.442956    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws" event={"ID":"0164ec25-f653-42a1-9860-8470c532ee23","Type":"ContainerStarted","Data":"a2a146267fe3a32eacacdc7ecd555f908d4a51d7eb8144f33e95b2d03ab7b666"}
Nov 01 21:30:49.443055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.442983    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:30:49.722166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.722112    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz" podStartSLOduration=2.722096274 podStartE2EDuration="2.722096274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:49.605919576 +0000 UTC m=+5302.990817165" watchObservedRunningTime="2026-11-01 21:30:49.722096274 +0000 UTC m=+5303.106993863"
Nov 01 21:30:49.723437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:49.723406    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws" podStartSLOduration=2.7233971390000002 podStartE2EDuration="2.723397139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:30:49.719826316 +0000 UTC m=+5303.104723906" watchObservedRunningTime="2026-11-01 21:30:49.723397139 +0000 UTC m=+5303.108294728"
Nov 01 21:30:50.540324 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.540272    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" podUID="c758dba3-c56f-4436-af91-e25f42d98603" containerName="payload-processing" containerID="cri-o://28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee" gracePeriod=25
Nov 01 21:30:50.798969 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.798899    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 21:30:50.929423 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.929390    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\" (UniqueName: \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\") pod \"c758dba3-c56f-4436-af91-e25f42d98603\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") "
Nov 01 21:30:50.929658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.929482    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\") pod \"c758dba3-c56f-4436-af91-e25f42d98603\" (UID: \"c758dba3-c56f-4436-af91-e25f42d98603\") "
Nov 01 21:30:50.929858 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.929828    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume" pod "c758dba3-c56f-4436-af91-e25f42d98603" (UID: "c758dba3-c56f-4436-af91-e25f42d98603"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:30:50.930035 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.929998    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c758dba3-c56f-4436-af91-e25f42d98603-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:50.932163 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:50.932137    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf" pod "c758dba3-c56f-4436-af91-e25f42d98603" (UID: "c758dba3-c56f-4436-af91-e25f42d98603"). InnerVolumeSpecName "kube-api-access-v9lhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:30:51.031332 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.031287    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9lhf\" (UniqueName: \"kubernetes.io/projected/c758dba3-c56f-4436-af91-e25f42d98603-kube-api-access-v9lhf\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:30:51.456897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.455719    5066 generic.go:356] "Generic (PLEG): container finished" podID="c758dba3-c56f-4436-af91-e25f42d98603" containerID="28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee" exitCode=0
Nov 01 21:30:51.456897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.455775    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"
Nov 01 21:30:51.456897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.455916    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" event={"ID":"c758dba3-c56f-4436-af91-e25f42d98603","Type":"ContainerDied","Data":"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"}
Nov 01 21:30:51.456897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.455956    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fhnrf" event={"ID":"c758dba3-c56f-4436-af91-e25f42d98603","Type":"ContainerDied","Data":"07e1a4b3e70b09e12b7d1fe2c0e3f4c476addbb41408342e04c2fa914e9d1f7b"}
Nov 01 21:30:51.456897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.455986    5066 scope.go:122] "RemoveContainer" containerID="28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"
Nov 01 21:30:51.467749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.467724    5066 scope.go:122] "RemoveContainer" containerID="28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"
Nov 01 21:30:51.468032 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:51.468009    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee\": container with ID starting with 28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee not found: ID does not exist" containerID="28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"
Nov 01 21:30:51.468106 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.468039    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee"} err="failed to get container status \"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee\": rpc error: code = NotFound desc = could not find container \"28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee\": container with ID starting with 28a3ce20767ca0b48acf115c47764cfaf37a42149bc2f1420137115a4c89dfee not found: ID does not exist"
Nov 01 21:30:51.624112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.624080    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 21:30:51.628854 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:30:51.628821    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:30:51.674942 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:51.674909    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fhnrf"]
Nov 01 21:30:52.729663 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:30:52.729630    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c758dba3-c56f-4436-af91-e25f42d98603" path="/var/lib/kubelet/pods/c758dba3-c56f-4436-af91-e25f42d98603/volumes"
Nov 01 21:31:00.451360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:00.451325    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7f5ccd4859-xftxz"
Nov 01 21:31:00.451835 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:00.451391    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7d98d48688-6pcws"
Nov 01 21:31:00.621177 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:00.621143    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:31:00.706061 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:00.705976    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:31:05.621443 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:05.621397    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" podUID="1ec55dba-a534-4d06-9ab8-071724ec0641" containerName="payload-pre-processing" containerID="cri-o://e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d" gracePeriod=25
Nov 01 21:31:05.706646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:05.706602    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" podUID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" containerName="payload-processing" containerID="cri-o://56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08" gracePeriod=25
Nov 01 21:31:05.967902 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:05.967873    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:31:05.995340 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:05.995314    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:31:06.077389 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077356    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\") pod \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") "
Nov 01 21:31:06.077595 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077456    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\" (UniqueName: \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\") pod \"1ec55dba-a534-4d06-9ab8-071724ec0641\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") "
Nov 01 21:31:06.077640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077596    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\") pod \"1ec55dba-a534-4d06-9ab8-071724ec0641\" (UID: \"1ec55dba-a534-4d06-9ab8-071724ec0641\") "
Nov 01 21:31:06.077640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077627    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\" (UniqueName: \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\") pod \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\" (UID: \"e60c20a3-7577-4dc8-83a7-b7668d8bac05\") "
Nov 01 21:31:06.077894 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077871    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume" pod "e60c20a3-7577-4dc8-83a7-b7668d8bac05" (UID: "e60c20a3-7577-4dc8-83a7-b7668d8bac05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:31:06.077985 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077908    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume" pod "1ec55dba-a534-4d06-9ab8-071724ec0641" (UID: "1ec55dba-a534-4d06-9ab8-071724ec0641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:31:06.078037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.077993    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec55dba-a534-4d06-9ab8-071724ec0641-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:06.078037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.078004    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60c20a3-7577-4dc8-83a7-b7668d8bac05-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:06.079610 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.079591    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw" pod "1ec55dba-a534-4d06-9ab8-071724ec0641" (UID: "1ec55dba-a534-4d06-9ab8-071724ec0641"). InnerVolumeSpecName "kube-api-access-stkgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:31:06.079688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.079600    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl" pod "e60c20a3-7577-4dc8-83a7-b7668d8bac05" (UID: "e60c20a3-7577-4dc8-83a7-b7668d8bac05"). InnerVolumeSpecName "kube-api-access-26lxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:31:06.178884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.178788    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stkgw\" (UniqueName: \"kubernetes.io/projected/1ec55dba-a534-4d06-9ab8-071724ec0641-kube-api-access-stkgw\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:06.178884 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.178825    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lxl\" (UniqueName: \"kubernetes.io/projected/e60c20a3-7577-4dc8-83a7-b7668d8bac05-kube-api-access-26lxl\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:31:06.541587 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.541548    5066 generic.go:356] "Generic (PLEG): container finished" podID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" containerID="56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08" exitCode=0
Nov 01 21:31:06.541800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.541601    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4"
Nov 01 21:31:06.541800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.541655    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" event={"ID":"e60c20a3-7577-4dc8-83a7-b7668d8bac05","Type":"ContainerDied","Data":"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"}
Nov 01 21:31:06.541800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.541718    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-56645ccfc8-m6sg4" event={"ID":"e60c20a3-7577-4dc8-83a7-b7668d8bac05","Type":"ContainerDied","Data":"6756f22742d3651aa1f79f39de05e0d2b6e4a3d70bfa688f84943b8558dbdd90"}
Nov 01 21:31:06.541800 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.541751    5066 scope.go:122] "RemoveContainer" containerID="56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"
Nov 01 21:31:06.543172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.543146    5066 generic.go:356] "Generic (PLEG): container finished" podID="1ec55dba-a534-4d06-9ab8-071724ec0641" containerID="e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d" exitCode=0
Nov 01 21:31:06.543299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.543187    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"
Nov 01 21:31:06.543299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.543222    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" event={"ID":"1ec55dba-a534-4d06-9ab8-071724ec0641","Type":"ContainerDied","Data":"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"}
Nov 01 21:31:06.543299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.543258    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh" event={"ID":"1ec55dba-a534-4d06-9ab8-071724ec0641","Type":"ContainerDied","Data":"cb9438fa43603c4415d33ab363a77de4bd2e73a243df49852f74473fc658d1aa"}
Nov 01 21:31:06.551580 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.551564    5066 scope.go:122] "RemoveContainer" containerID="56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"
Nov 01 21:31:06.551861 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:31:06.551842    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08\": container with ID starting with 56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08 not found: ID does not exist" containerID="56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"
Nov 01 21:31:06.551955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.551867    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08"} err="failed to get container status \"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08\": rpc error: code = NotFound desc = could not find container \"56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08\": container with ID starting with 56fb9023fdd041c36544906a5b16167a5a072013dd44b1464323144bcca33a08 not found: ID does not exist"
Nov 01 21:31:06.551955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.551893    5066 scope.go:122] "RemoveContainer" containerID="e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"
Nov 01 21:31:06.560390 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.560368    5066 scope.go:122] "RemoveContainer" containerID="e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"
Nov 01 21:31:06.560651 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:31:06.560630    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d\": container with ID starting with e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d not found: ID does not exist" containerID="e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"
Nov 01 21:31:06.560700 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.560658    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d"} err="failed to get container status \"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d\": rpc error: code = NotFound desc = could not find container \"e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d\": container with ID starting with e4c4f949ebccb287d23f85e91a36882728133594c4c4af91f126cca4e18e798d not found: ID does not exist"
Nov 01 21:31:06.708825 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.708793    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:31:06.759391 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.759357    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-54c4dc45d9-zw2xh"]
Nov 01 21:31:06.802817 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.802721    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:31:06.803166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803151    5066 cpu_manager.go:414] "containerMap: removing container" podUID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" containerName="payload-processing"
Nov 01 21:31:06.803166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803165    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" containerName="payload-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803178    5066 cpu_manager.go:414] "containerMap: removing container" podUID="c758dba3-c56f-4436-af91-e25f42d98603" containerName="payload-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803183    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c758dba3-c56f-4436-af91-e25f42d98603" containerName="payload-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803196    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1ec55dba-a534-4d06-9ab8-071724ec0641" containerName="payload-pre-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803202    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec55dba-a534-4d06-9ab8-071724ec0641" containerName="payload-pre-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803292    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" containerName="payload-processing"
Nov 01 21:31:06.803300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803300    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="c758dba3-c56f-4436-af91-e25f42d98603" containerName="payload-processing"
Nov 01 21:31:06.803640 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.803307    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec55dba-a534-4d06-9ab8-071724ec0641" containerName="payload-pre-processing"
Nov 01 21:31:06.807735 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.807712    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.821943 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.821914    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:31:06.827170 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.827139    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-56645ccfc8-m6sg4"]
Nov 01 21:31:06.860731 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.860648    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 21:31:06.860731 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.860648    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 01 21:31:06.860980 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.860777    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 21:31:06.860980 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.860785    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 21:31:06.860980 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.860894    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zhqtr\""
Nov 01 21:31:06.862288 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.862264    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:31:06.868432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.868407    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 21:31:06.886762 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.886727    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.886949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.886783    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.887028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.886978    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdz9r\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.887028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.887008    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.988133 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.988083    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.988133 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.988139    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.988367 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.988237    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdz9r\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.988367 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.988256    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.988936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.988908    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.989062 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.989005    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:06.990596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:06.990573    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:07.045360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:07.045327    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdz9r\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\") pod \"payload-processing-external-model-5bcd7997-tp96f\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:07.121341 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:07.121259    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:07.515240 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:07.515208    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:31:07.515947 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:31:07.515921    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800c6bb7_7d57_4652_9122_f554a7693a43.slice/crio-c105a59ec77dde2797c89be8dc0e6c2d259adc70d77c06d2c23becf310542eeb WatchSource:0}: Error finding container c105a59ec77dde2797c89be8dc0e6c2d259adc70d77c06d2c23becf310542eeb: Status 404 returned error can't find the container with id c105a59ec77dde2797c89be8dc0e6c2d259adc70d77c06d2c23becf310542eeb
Nov 01 21:31:07.550133 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:07.550102    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" event={"ID":"800c6bb7-7d57-4652-9122-f554a7693a43","Type":"ContainerStarted","Data":"c105a59ec77dde2797c89be8dc0e6c2d259adc70d77c06d2c23becf310542eeb"}
Nov 01 21:31:08.562172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:08.562130    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" event={"ID":"800c6bb7-7d57-4652-9122-f554a7693a43","Type":"ContainerStarted","Data":"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"}
Nov 01 21:31:08.562693 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:08.562379    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:08.679917 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:08.679847    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" podStartSLOduration=2.6798270239999997 podStartE2EDuration="2.679827024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:31:08.676702943 +0000 UTC m=+5322.061600533" watchObservedRunningTime="2026-11-01 21:31:08.679827024 +0000 UTC m=+5322.064724614"
Nov 01 21:31:08.733933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:08.733884    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec55dba-a534-4d06-9ab8-071724ec0641" path="/var/lib/kubelet/pods/1ec55dba-a534-4d06-9ab8-071724ec0641/volumes"
Nov 01 21:31:08.734476 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:08.734450    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60c20a3-7577-4dc8-83a7-b7668d8bac05" path="/var/lib/kubelet/pods/e60c20a3-7577-4dc8-83a7-b7668d8bac05/volumes"
Nov 01 21:31:19.574037 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:19.574006    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:31:21.632660 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:31:21.632619    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:31:28.378065 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:31:28.378035    5066 scope.go:122] "RemoveContainer" containerID="b62be6735888e17fd5483896066b05fb990bbfb66bab00635ce164d7681be284"
Nov 01 21:31:51.627614 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:31:51.627573    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:32:21.628264 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:32:21.628222    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:32:27.555580 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:32:27.555552    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:32:27.618604 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:32:27.618580    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:32:28.411007 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:32:28.410978    5066 scope.go:122] "RemoveContainer" containerID="17e3ad0f517c6117b365660690bfc96b670e4fabb975c13f95d68b834ccc6916"
Nov 01 21:32:51.627367 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:32:51.627325    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:33:05.205685 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:05.205646    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:33:10.207574 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:10.206566    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" podUID="800c6bb7-7d57-4652-9122-f554a7693a43" containerName="payload-processing" containerID="cri-o://4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb" gracePeriod=25
Nov 01 21:33:11.176140 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.175801    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:33:11.298849 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.297810    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\") pod \"800c6bb7-7d57-4652-9122-f554a7693a43\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") "
Nov 01 21:33:11.298849 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.297920    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\") pod \"800c6bb7-7d57-4652-9122-f554a7693a43\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") "
Nov 01 21:33:11.298849 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.297993    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\") pod \"800c6bb7-7d57-4652-9122-f554a7693a43\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") "
Nov 01 21:33:11.298849 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.298116    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\") pod \"800c6bb7-7d57-4652-9122-f554a7693a43\" (UID: \"800c6bb7-7d57-4652-9122-f554a7693a43\") "
Nov 01 21:33:11.300394 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.300347    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume" pod "800c6bb7-7d57-4652-9122-f554a7693a43" (UID: "800c6bb7-7d57-4652-9122-f554a7693a43"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:33:11.300991 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.300951    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume" pod "800c6bb7-7d57-4652-9122-f554a7693a43" (UID: "800c6bb7-7d57-4652-9122-f554a7693a43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:33:11.397759 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.397681    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r" pod "800c6bb7-7d57-4652-9122-f554a7693a43" (UID: "800c6bb7-7d57-4652-9122-f554a7693a43"). InnerVolumeSpecName "kube-api-access-sdz9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:33:11.423718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.422774    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdz9r\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-kube-api-access-sdz9r\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:11.423718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.422809    5066 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-routing-overlay-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:11.423718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.422825    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/800c6bb7-7d57-4652-9122-f554a7693a43-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:11.425611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.424536    5066 generic.go:356] "Generic (PLEG): container finished" podID="800c6bb7-7d57-4652-9122-f554a7693a43" containerID="4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb" exitCode=0
Nov 01 21:33:11.425611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.424731    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" event={"ID":"800c6bb7-7d57-4652-9122-f554a7693a43","Type":"ContainerDied","Data":"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"}
Nov 01 21:33:11.425611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.424786    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f" event={"ID":"800c6bb7-7d57-4652-9122-f554a7693a43","Type":"ContainerDied","Data":"c105a59ec77dde2797c89be8dc0e6c2d259adc70d77c06d2c23becf310542eeb"}
Nov 01 21:33:11.425611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.424818    5066 scope.go:122] "RemoveContainer" containerID="4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"
Nov 01 21:33:11.425611 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.424997    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"
Nov 01 21:33:11.454887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.454826    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials" pod "800c6bb7-7d57-4652-9122-f554a7693a43" (UID: "800c6bb7-7d57-4652-9122-f554a7693a43"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:33:11.519758 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.510062    5066 scope.go:122] "RemoveContainer" containerID="4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"
Nov 01 21:33:11.519758 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:33:11.510485    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb\": container with ID starting with 4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb not found: ID does not exist" containerID="4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"
Nov 01 21:33:11.519758 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.519665    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb"} err="failed to get container status \"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb\": rpc error: code = NotFound desc = could not find container \"4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb\": container with ID starting with 4a02148be005026dc8e423870f6eddedad80aec380a6abfae75128b22851b6bb not found: ID does not exist"
Nov 01 21:33:11.528204 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.528150    5066 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/800c6bb7-7d57-4652-9122-f554a7693a43-provider-credentials\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:33:11.847229 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.847188    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:33:11.853197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:11.853166    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-tp96f"]
Nov 01 21:33:12.730567 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:12.730530    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800c6bb7-7d57-4652-9122-f554a7693a43" path="/var/lib/kubelet/pods/800c6bb7-7d57-4652-9122-f554a7693a43/volumes"
Nov 01 21:33:21.773525 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:33:21.770584    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:33:47.045519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:47.044617    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:33:47.045519 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:47.044711    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:33:47.258662 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:47.258615    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:33:47.258877 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:33:47.258691    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:33:51.627741 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:33:51.627702    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:34:21.628720 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:34:21.628679    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:34:51.627802 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:34:51.627767    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:35:21.629634 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:35:21.629599    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:35:47.038087 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:35:47.038026    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:35:47.038843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:35:47.038807    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:35:47.266910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:35:47.263610    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:35:47.266910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:35:47.263690    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:35:51.628194 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:35:51.628152    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:36:21.627393 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:36:21.627309    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:36:51.627293 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:36:51.627256    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:37:02.041999 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.041943    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pgrt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:37:02.042857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.042808    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:37:02.256726 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.256683    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vdzv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:37:02.257001 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.256970    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:37:02.942783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.942742    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ljwmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:37:02.943109 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:02.943080    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:37:21.627758 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:37:21.627719    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:37:27.588785 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:27.588682    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:37:27.655585 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:37:27.655558    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:37:51.628751 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:37:51.628710    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:38:21.628867 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:38:21.628831    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:38:36.194660 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.194625    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:38:36.195907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.195878    5066 cpu_manager.go:414] "containerMap: removing container" podUID="800c6bb7-7d57-4652-9122-f554a7693a43" containerName="payload-processing"
Nov 01 21:38:36.196080 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.196062    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800c6bb7-7d57-4652-9122-f554a7693a43" containerName="payload-processing"
Nov 01 21:38:36.196311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.196295    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="800c6bb7-7d57-4652-9122-f554a7693a43" containerName="payload-processing"
Nov 01 21:38:36.207039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.207014    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.288850 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.288796    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c660c6-openshift-default-dockercfg-8jxmq\""
Nov 01 21:38:36.420271 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.420230    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.420590 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.420565    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.420751 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.420733    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.420910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.420893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.421039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.421022    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.421178 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.421159    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.421364 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.421344    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.421577 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.421555    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n446w\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.430039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.430010    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.430650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.429662    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.535887    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.535939    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.535985    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536020    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536071    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536099    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536147    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536204    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536281    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n446w\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.536999    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.537517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.537252    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.538294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.537889    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.559516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.538556    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.559516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.539329    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.596124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.595712    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.629225 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.621472    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.632476 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.632418    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n446w\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.658818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.658756    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\") pod \"e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") " pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:36.837790 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:36.837698    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:37.771406 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:37.771356    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:38:37.777787 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:37.777749    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:38:37.778014 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:37.777989    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:38:37.778169 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:37.778147    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:38:38.781731 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:38.781694    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerStarted","Data":"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"}
Nov 01 21:38:38.782195 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:38.781741    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerStarted","Data":"a4441045da983403f9b44144a234cd6c6bca4d765e962dd9766c30f54d543c9a"}
Nov 01 21:38:38.838393 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:38.838368    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:38.844217 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:38.844193    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:38.951910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:38.951865    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podStartSLOduration=2.951851465 podStartE2EDuration="2.951851465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:38:38.948860374 +0000 UTC m=+5772.333757963" watchObservedRunningTime="2026-11-01 21:38:38.951851465 +0000 UTC m=+5772.336749052"
Nov 01 21:38:39.787006 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:39.786976    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:39.787897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:39.787873    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:38:45.311895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.311858    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:38:45.316818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.316797    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.372475 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.372438    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:38:45.377641 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.377615    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.377806 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.377749    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.418109 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.418086    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c660c6\""
Nov 01 21:38:45.472986 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.472943    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:38:45.476925 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.476903    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.478335 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.478317    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.478421 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.478385    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.479173 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.479149    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.519876 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.519852    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:38:45.529687 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.529666    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-63745a57-dockercfg-gqsq7\""
Nov 01 21:38:45.535536 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.535514    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.579587 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.579518    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.579587 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.579570    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82c2m\" (UniqueName: \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.632002 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.631973    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:45.681098 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.681064    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.681230 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.681127    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82c2m\" (UniqueName: \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.683453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.682290    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.738303 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.738272    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82c2m\" (UniqueName: \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:45.790285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:45.790260    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:46.005101 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.005069    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:38:46.005466 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:38:46.005441    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312cf434_c48d_4e55_b8bc_68dc59ef3aa1.slice/crio-2c33e018b1f3ed6cfb63f6e04d7460387e40b5e9b1475fd626fa5ca45deafa52 WatchSource:0}: Error finding container 2c33e018b1f3ed6cfb63f6e04d7460387e40b5e9b1475fd626fa5ca45deafa52: Status 404 returned error can't find the container with id 2c33e018b1f3ed6cfb63f6e04d7460387e40b5e9b1475fd626fa5ca45deafa52
Nov 01 21:38:46.165599 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.165571    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:38:46.165876 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:38:46.165847    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785466c6_78a9_4970_82ae_3efe14e727a5.slice/crio-81c1f8fe205c3ba15d6d59b64caf9113daf2103b1f36e2331bcdf80f5c9f0878 WatchSource:0}: Error finding container 81c1f8fe205c3ba15d6d59b64caf9113daf2103b1f36e2331bcdf80f5c9f0878: Status 404 returned error can't find the container with id 81c1f8fe205c3ba15d6d59b64caf9113daf2103b1f36e2331bcdf80f5c9f0878
Nov 01 21:38:46.825846 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.825799    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" event={"ID":"312cf434-c48d-4e55-b8bc-68dc59ef3aa1","Type":"ContainerStarted","Data":"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"}
Nov 01 21:38:46.826243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.825861    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" event={"ID":"312cf434-c48d-4e55-b8bc-68dc59ef3aa1","Type":"ContainerStarted","Data":"2c33e018b1f3ed6cfb63f6e04d7460387e40b5e9b1475fd626fa5ca45deafa52"}
Nov 01 21:38:46.826243 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.825888    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:38:46.827790 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.827766    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" event={"ID":"785466c6-78a9-4970-82ae-3efe14e727a5","Type":"ContainerStarted","Data":"e7048a4c4cabe177021aa90a60dc72b95a55019c69309dd9f04607c758c88679"}
Nov 01 21:38:46.827942 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.827799    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" event={"ID":"785466c6-78a9-4970-82ae-3efe14e727a5","Type":"ContainerStarted","Data":"81c1f8fe205c3ba15d6d59b64caf9113daf2103b1f36e2331bcdf80f5c9f0878"}
Nov 01 21:38:46.827942 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.827842    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:46.939300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:46.939252    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" podStartSLOduration=1.939239053 podStartE2EDuration="1.939239053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:38:46.938916395 +0000 UTC m=+5780.323813985" watchObservedRunningTime="2026-11-01 21:38:46.939239053 +0000 UTC m=+5780.324136644"
Nov 01 21:38:47.054331 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.054279    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" podStartSLOduration=2.054262061 podStartE2EDuration="2.054262061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:38:47.05230468 +0000 UTC m=+5780.437202270" watchObservedRunningTime="2026-11-01 21:38:47.054262061 +0000 UTC m=+5780.439159653"
Nov 01 21:38:47.563558 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.563520    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:38:47.567279 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.567252    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.623717 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.623683    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:38:47.678541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.671047    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:38:47.678541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.671651    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c660c6-serving-cert\""
Nov 01 21:38:47.678541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.671764    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:38:47.678541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.671825    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:38:47.678541 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.671899    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:38:47.699943 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.699900    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.700124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.700069    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.700192 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.700117    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66qv\" (UniqueName: \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.700192 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.700171    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.801180 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.801139    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.801388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.801232    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.801472 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.801402    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66qv\" (UniqueName: \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.801472 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.801450    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.802266 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:38:47.802236    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-serving-cert: secret "maas-api-e2e-worker-single-main-c660c6-serving-cert" not found
Nov 01 21:38:47.802439 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.802413    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.802579 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:38:47.802563    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls podName:153be7fd-039a-4186-b351-f7ca4325d5c9 nodeName:}" failed. No retries permitted until 2026-11-01 21:38:48.302432993 +0000 UTC m=+5781.687330574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls") pod "maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" (UID: "153be7fd-039a-4186-b351-f7ca4325d5c9") : secret "maas-api-e2e-worker-single-main-c660c6-serving-cert" not found
Nov 01 21:38:47.806322 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.806275    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:47.860552 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:47.860445    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66qv\" (UniqueName: \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:48.306423 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:48.306391    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:48.308576 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:48.308548    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:48.481804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:48.481772    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:48.868095 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:38:48.868055    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153be7fd_039a_4186_b351_f7ca4325d5c9.slice/crio-f06e45153a49b36222c52feb2eeb1f6c7d5940191c448a712b5ff9696dfbbd38 WatchSource:0}: Error finding container f06e45153a49b36222c52feb2eeb1f6c7d5940191c448a712b5ff9696dfbbd38: Status 404 returned error can't find the container with id f06e45153a49b36222c52feb2eeb1f6c7d5940191c448a712b5ff9696dfbbd38
Nov 01 21:38:48.868533 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:48.868216    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:38:48.869474 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:48.869456    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:38:49.848539 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:49.848459    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" event={"ID":"153be7fd-039a-4186-b351-f7ca4325d5c9","Type":"ContainerStarted","Data":"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"}
Nov 01 21:38:49.848701 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:49.848544    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" event={"ID":"153be7fd-039a-4186-b351-f7ca4325d5c9","Type":"ContainerStarted","Data":"f06e45153a49b36222c52feb2eeb1f6c7d5940191c448a712b5ff9696dfbbd38"}
Nov 01 21:38:49.848701 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:49.848582    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:49.997316 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:49.997217    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" podStartSLOduration=2.119947013 podStartE2EDuration="2.997205294s" totalImagesPullingTime="877.258281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:38:49.99354738 +0000 UTC m=+5783.378444960" watchObservedRunningTime="2026-11-01 21:38:49.997205294 +0000 UTC m=+5783.382102884"
Nov 01 21:38:51.627871 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:38:51.627837    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:38:55.859649 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:55.859617    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:38:57.837001 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:57.836970    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:38:57.837416 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:38:57.837031    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:39:04.942267 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.942232    5066 generic.go:356] "Generic (PLEG): container finished" podID="832410b1-6377-434a-91db-971698485dd4" containerID="427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341" exitCode=0
Nov 01 21:39:04.942718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.942294    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerDied","Data":"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"}
Nov 01 21:39:04.942718 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.942660    5066 scope.go:122] "RemoveContainer" containerID="427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"
Nov 01 21:39:04.943804 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.943776    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:39:04.943876 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.943844    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:39:04.943913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:04.943893    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:39:05.949061 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:05.949021    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerStarted","Data":"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"}
Nov 01 21:39:06.838586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:06.838552    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:39:06.839414 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:06.839391    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 21:39:06.839461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:06.839444    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 21:39:07.839232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:07.839156    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 21:39:07.839232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:07.839203    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 21:39:08.838618 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:08.838582    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 21:39:08.838826 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:08.838647    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 21:39:09.787813 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:09.787757    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:39:09.838748 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:09.838717    5066 patch_prober.go:29] interesting pod/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 21:39:09.838898 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:09.838762    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 21:39:10.924138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:10.924108    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:39:10.927090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:39:10.927064    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:39:21.628774 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:39:21.628735    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:39:51.628553 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:39:51.628516    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:40:11.894501 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:11.894459    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:40:11.894923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:11.894652    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" podUID="153be7fd-039a-4186-b351-f7ca4325d5c9" containerName="maas-api" containerID="cri-o://4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae" gracePeriod=30
Nov 01 21:40:12.362694 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:12.362646    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:40:12.415997 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:12.415957    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:40:13.153623 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.153594    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:40:13.295932 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.295841    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") pod \"153be7fd-039a-4186-b351-f7ca4325d5c9\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") "
Nov 01 21:40:13.295932 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.295877    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\" (UniqueName: \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\") pod \"153be7fd-039a-4186-b351-f7ca4325d5c9\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") "
Nov 01 21:40:13.295932 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.295898    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\") pod \"153be7fd-039a-4186-b351-f7ca4325d5c9\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") "
Nov 01 21:40:13.296241 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.295969    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\") pod \"153be7fd-039a-4186-b351-f7ca4325d5c9\" (UID: \"153be7fd-039a-4186-b351-f7ca4325d5c9\") "
Nov 01 21:40:13.296389 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.296357    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca" pod "153be7fd-039a-4186-b351-f7ca4325d5c9" (UID: "153be7fd-039a-4186-b351-f7ca4325d5c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:40:13.298151 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.298127    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls" pod "153be7fd-039a-4186-b351-f7ca4325d5c9" (UID: "153be7fd-039a-4186-b351-f7ca4325d5c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:40:13.298473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.298451    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs" pod "153be7fd-039a-4186-b351-f7ca4325d5c9" (UID: "153be7fd-039a-4186-b351-f7ca4325d5c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:40:13.298571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.298471    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv" pod "153be7fd-039a-4186-b351-f7ca4325d5c9" (UID: "153be7fd-039a-4186-b351-f7ca4325d5c9"). InnerVolumeSpecName "kube-api-access-q66qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:40:13.349850 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.349819    5066 generic.go:356] "Generic (PLEG): container finished" podID="153be7fd-039a-4186-b351-f7ca4325d5c9" containerID="4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae" exitCode=0
Nov 01 21:40:13.349989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.349862    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"
Nov 01 21:40:13.349989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.349898    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" event={"ID":"153be7fd-039a-4186-b351-f7ca4325d5c9","Type":"ContainerDied","Data":"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"}
Nov 01 21:40:13.349989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.349949    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv" event={"ID":"153be7fd-039a-4186-b351-f7ca4325d5c9","Type":"ContainerDied","Data":"f06e45153a49b36222c52feb2eeb1f6c7d5940191c448a712b5ff9696dfbbd38"}
Nov 01 21:40:13.349989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.349973    5066 scope.go:122] "RemoveContainer" containerID="4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"
Nov 01 21:40:13.361922 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.361905    5066 scope.go:122] "RemoveContainer" containerID="4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"
Nov 01 21:40:13.362235 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:13.362210    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae\": container with ID starting with 4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae not found: ID does not exist" containerID="4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"
Nov 01 21:40:13.362300 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.362244    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae"} err="failed to get container status \"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae\": rpc error: code = NotFound desc = could not find container \"4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae\": container with ID starting with 4a9c7fddfdb3b52ec2f71bb232cd274f2e05c604e415ed1140fa934b0b7ecbae not found: ID does not exist"
Nov 01 21:40:13.396834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.396806    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:13.396834 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.396830    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66qv\" (UniqueName: \"kubernetes.io/projected/153be7fd-039a-4186-b351-f7ca4325d5c9-kube-api-access-q66qv\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:13.396988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.396840    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153be7fd-039a-4186-b351-f7ca4325d5c9-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:13.396988 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.396848    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153be7fd-039a-4186-b351-f7ca4325d5c9-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:13.516909 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.516872    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:40:13.520278 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:13.520260    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-zrqqv"]
Nov 01 21:40:14.730867 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:14.730834    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153be7fd-039a-4186-b351-f7ca4325d5c9" path="/var/lib/kubelet/pods/153be7fd-039a-4186-b351-f7ca4325d5c9/volumes"
Nov 01 21:40:17.363189 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:17.363122    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing" containerID="cri-o://e7048a4c4cabe177021aa90a60dc72b95a55019c69309dd9f04607c758c88679" gracePeriod=25
Nov 01 21:40:17.416647 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:17.416590    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing" containerID="cri-o://ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323" gracePeriod=25
Nov 01 21:40:17.835576 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:17.835536    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:40:17.835749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:17.835641    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:40:21.628830 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:21.628797    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:40:27.835959 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:27.835918    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:40:27.836415 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:27.835925    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 21:40:37.488318 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.488291    5066 generic.go:356] "Generic (PLEG): container finished" podID="785466c6-78a9-4970-82ae-3efe14e727a5" containerID="e7048a4c4cabe177021aa90a60dc72b95a55019c69309dd9f04607c758c88679" exitCode=0
Nov 01 21:40:37.488752 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.488365    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" event={"ID":"785466c6-78a9-4970-82ae-3efe14e727a5","Type":"ContainerDied","Data":"e7048a4c4cabe177021aa90a60dc72b95a55019c69309dd9f04607c758c88679"}
Nov 01 21:40:37.692801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.692778    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:40:37.710964 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.710943    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:40:37.733429 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.733405    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\") pod \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") "
Nov 01 21:40:37.733551 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.733435    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\" (UniqueName: \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\") pod \"785466c6-78a9-4970-82ae-3efe14e727a5\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") "
Nov 01 21:40:37.733551 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.733474    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\") pod \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\" (UID: \"312cf434-c48d-4e55-b8bc-68dc59ef3aa1\") "
Nov 01 21:40:37.733678 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.733660    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\") pod \"785466c6-78a9-4970-82ae-3efe14e727a5\" (UID: \"785466c6-78a9-4970-82ae-3efe14e727a5\") "
Nov 01 21:40:37.733919 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.733834    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume" pod "312cf434-c48d-4e55-b8bc-68dc59ef3aa1" (UID: "312cf434-c48d-4e55-b8bc-68dc59ef3aa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:40:37.734027 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.734002    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume" pod "785466c6-78a9-4970-82ae-3efe14e727a5" (UID: "785466c6-78a9-4970-82ae-3efe14e727a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:40:37.734090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.734067    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:37.735360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.735341    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8" pod "312cf434-c48d-4e55-b8bc-68dc59ef3aa1" (UID: "312cf434-c48d-4e55-b8bc-68dc59ef3aa1"). InnerVolumeSpecName "kube-api-access-sbmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:40:37.735428 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.735411    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m" pod "785466c6-78a9-4970-82ae-3efe14e727a5" (UID: "785466c6-78a9-4970-82ae-3efe14e727a5"). InnerVolumeSpecName "kube-api-access-82c2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:40:37.834965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.834899    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmf8\" (UniqueName: \"kubernetes.io/projected/312cf434-c48d-4e55-b8bc-68dc59ef3aa1-kube-api-access-sbmf8\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:37.834965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.834924    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82c2m\" (UniqueName: \"kubernetes.io/projected/785466c6-78a9-4970-82ae-3efe14e727a5-kube-api-access-82c2m\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:37.834965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:37.834934    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785466c6-78a9-4970-82ae-3efe14e727a5-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:40:38.494901 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.494870    5066 generic.go:356] "Generic (PLEG): container finished" podID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerID="ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323" exitCode=0
Nov 01 21:40:38.495330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.494914    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"
Nov 01 21:40:38.495330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.494956    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" event={"ID":"312cf434-c48d-4e55-b8bc-68dc59ef3aa1","Type":"ContainerDied","Data":"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"}
Nov 01 21:40:38.495330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.495010    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh" event={"ID":"312cf434-c48d-4e55-b8bc-68dc59ef3aa1","Type":"ContainerDied","Data":"2c33e018b1f3ed6cfb63f6e04d7460387e40b5e9b1475fd626fa5ca45deafa52"}
Nov 01 21:40:38.495330 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.495042    5066 scope.go:122] "RemoveContainer" containerID="ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"
Nov 01 21:40:38.496418 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.496401    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"
Nov 01 21:40:38.496565 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.496422    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz" event={"ID":"785466c6-78a9-4970-82ae-3efe14e727a5","Type":"ContainerDied","Data":"81c1f8fe205c3ba15d6d59b64caf9113daf2103b1f36e2331bcdf80f5c9f0878"}
Nov 01 21:40:38.504981 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.504966    5066 scope.go:122] "RemoveContainer" containerID="ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"
Nov 01 21:40:38.505232 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:38.505209    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323\": container with ID starting with ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323 not found: ID does not exist" containerID="ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"
Nov 01 21:40:38.505311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.505244    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323"} err="failed to get container status \"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323\": rpc error: code = NotFound desc = could not find container \"ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323\": container with ID starting with ed140f3ebd7957a46f763f7defbc0f779cc8af6f17af819732b2a42c46d8f323 not found: ID does not exist"
Nov 01 21:40:38.505311 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.505268    5066 scope.go:122] "RemoveContainer" containerID="e7048a4c4cabe177021aa90a60dc72b95a55019c69309dd9f04607c758c88679"
Nov 01 21:40:38.662568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.662485    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:40:38.721950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.721919    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fjspmh"]
Nov 01 21:40:38.729922 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.729897    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" path="/var/lib/kubelet/pods/312cf434-c48d-4e55-b8bc-68dc59ef3aa1/volumes"
Nov 01 21:40:38.822632 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.822606    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:40:38.825889 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:38.825868    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ff874qz"]
Nov 01 21:40:40.729752 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:40.729721    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" path="/var/lib/kubelet/pods/785466c6-78a9-4970-82ae-3efe14e727a5/volumes"
Nov 01 21:40:48.855760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.855726    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:40:48.856154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856141    5066 cpu_manager.go:414] "containerMap: removing container" podUID="153be7fd-039a-4186-b351-f7ca4325d5c9" containerName="maas-api"
Nov 01 21:40:48.856198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856155    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153be7fd-039a-4186-b351-f7ca4325d5c9" containerName="maas-api"
Nov 01 21:40:48.856198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856166    5066 cpu_manager.go:414] "containerMap: removing container" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing"
Nov 01 21:40:48.856198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856171    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing"
Nov 01 21:40:48.856198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856183    5066 cpu_manager.go:414] "containerMap: removing container" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing"
Nov 01 21:40:48.856198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856189    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing"
Nov 01 21:40:48.856355 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856278    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="153be7fd-039a-4186-b351-f7ca4325d5c9" containerName="maas-api"
Nov 01 21:40:48.856355 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856287    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="312cf434-c48d-4e55-b8bc-68dc59ef3aa1" containerName="payload-pre-processing"
Nov 01 21:40:48.856355 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.856293    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="785466c6-78a9-4970-82ae-3efe14e727a5" containerName="payload-processing"
Nov 01 21:40:48.859376 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.859361    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:48.917941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.917912    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:40:48.934402 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.934372    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:48.934591 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.934474    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:48.934591 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.934513    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:48.934591 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.934540    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g978q\" (UniqueName: \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:48.962431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.962400    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c660c6-serving-cert\""
Nov 01 21:40:48.962431 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.962410    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:40:48.964933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.964901    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:40:48.965039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.964933    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:40:48.965039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:48.964934    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:40:49.035651 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.035617    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.035810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.035664    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.035810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.035708    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g978q\" (UniqueName: \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.035936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.035827    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.035999 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:49.035954    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-serving-cert: secret "maas-api-e2e-worker-single-main-c660c6-serving-cert" not found
Nov 01 21:40:49.036056 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:49.036024    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls podName:71386dbc-7b7f-4395-98f0-cf4fabe8fa5c nodeName:}" failed. No retries permitted until 2026-11-01 21:40:49.536003573 +0000 UTC m=+5902.920901155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls") pod "maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" (UID: "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c") : secret "maas-api-e2e-worker-single-main-c660c6-serving-cert" not found
Nov 01 21:40:49.036385 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.036356    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.038127 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.038104    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.100908 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.100879    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g978q\" (UniqueName: \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.540965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.540931    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.543425 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.543398    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:49.772961 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:49.772931    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:50.146061 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.146030    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:40:50.146881 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:40:50.146849    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71386dbc_7b7f_4395_98f0_cf4fabe8fa5c.slice/crio-347f68fd6d5c006cdb3e3418dc1821cfd3024ab4249100364dfa4debfe86d100 WatchSource:0}: Error finding container 347f68fd6d5c006cdb3e3418dc1821cfd3024ab4249100364dfa4debfe86d100: Status 404 returned error can't find the container with id 347f68fd6d5c006cdb3e3418dc1821cfd3024ab4249100364dfa4debfe86d100
Nov 01 21:40:50.345545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.345474    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:40:50.350314 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.350293    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.429791 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.429703    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c660c6\""
Nov 01 21:40:50.429982 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.429964    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-63745a57-dockercfg-4cdv5\""
Nov 01 21:40:50.433631 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.433604    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:40:50.456299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.456260    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.456512 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.456307    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvd8w\" (UniqueName: \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.505808 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.505776    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:40:50.509401 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.509386    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.557922 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.557893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2w6\" (UniqueName: \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.558076 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.557957    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.558076 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.558007    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.558175 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.558080    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvd8w\" (UniqueName: \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.558749 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.558725    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.566184 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.566133    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:40:50.574248 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.574217    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" event={"ID":"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c","Type":"ContainerStarted","Data":"347f68fd6d5c006cdb3e3418dc1821cfd3024ab4249100364dfa4debfe86d100"}
Nov 01 21:40:50.616429 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.616397    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvd8w\" (UniqueName: \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\") pod \"payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.658809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.658780    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2w6\" (UniqueName: \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.658955 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.658840    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.659451 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.659433    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.664912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.664887    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:50.716537 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.716256    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2w6\" (UniqueName: \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\") pod \"payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:50.822845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:50.822812    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:51.052055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.052026    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:40:51.052211 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:40:51.052165    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1644ebd9_2ccb_4f93_acc1_1b5c6f647d33.slice/crio-2fb3ae74120179fbf0b26e3a58fdea04a4413696a119e3bfe70c84ab4a62f097 WatchSource:0}: Error finding container 2fb3ae74120179fbf0b26e3a58fdea04a4413696a119e3bfe70c84ab4a62f097: Status 404 returned error can't find the container with id 2fb3ae74120179fbf0b26e3a58fdea04a4413696a119e3bfe70c84ab4a62f097
Nov 01 21:40:51.196770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.196745    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:40:51.204711 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:40:51.204688    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2566f94d_8b18_4a0a_bb5a_45d7335af9f1.slice/crio-70d20c1ba6fbca360056fa304265103c227950a1c1e09580690081afe81dff6d WatchSource:0}: Error finding container 70d20c1ba6fbca360056fa304265103c227950a1c1e09580690081afe81dff6d: Status 404 returned error can't find the container with id 70d20c1ba6fbca360056fa304265103c227950a1c1e09580690081afe81dff6d
Nov 01 21:40:51.581295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.581257    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" event={"ID":"2566f94d-8b18-4a0a-bb5a-45d7335af9f1","Type":"ContainerStarted","Data":"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"}
Nov 01 21:40:51.581524 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.581315    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" event={"ID":"2566f94d-8b18-4a0a-bb5a-45d7335af9f1","Type":"ContainerStarted","Data":"70d20c1ba6fbca360056fa304265103c227950a1c1e09580690081afe81dff6d"}
Nov 01 21:40:51.581524 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.581376    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:40:51.582811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.582781    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" event={"ID":"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c","Type":"ContainerStarted","Data":"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"}
Nov 01 21:40:51.582906 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.582841    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:40:51.584200 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.584178    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" event={"ID":"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33","Type":"ContainerStarted","Data":"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"}
Nov 01 21:40:51.584294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.584218    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" event={"ID":"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33","Type":"ContainerStarted","Data":"2fb3ae74120179fbf0b26e3a58fdea04a4413696a119e3bfe70c84ab4a62f097"}
Nov 01 21:40:51.584294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.584241    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:40:51.627334 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:40:51.627306    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:40:51.698564 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.698513    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" podStartSLOduration=1.698500978 podStartE2EDuration="1.698500978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:40:51.695419295 +0000 UTC m=+5905.080316884" watchObservedRunningTime="2026-11-01 21:40:51.698500978 +0000 UTC m=+5905.083398559"
Nov 01 21:40:51.814013 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.813961    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" podStartSLOduration=2.791297326 podStartE2EDuration="3.813946298s" totalImagesPullingTime="1.022648972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:40:51.810469315 +0000 UTC m=+5905.195366905" watchObservedRunningTime="2026-11-01 21:40:51.813946298 +0000 UTC m=+5905.198843887"
Nov 01 21:40:51.924697 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:51.924610    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" podStartSLOduration=1.924596562 podStartE2EDuration="1.924596562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:40:51.923571722 +0000 UTC m=+5905.308469314" watchObservedRunningTime="2026-11-01 21:40:51.924596562 +0000 UTC m=+5905.309494186"
Nov 01 21:40:57.595890 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:40:57.595859    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:41:02.593668 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:02.593622    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:41:02.594157 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:02.594017    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:41:14.112586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:14.112530    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:41:14.220750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:14.220715    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:41:16.782933 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:16.782901    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:41:16.783347 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:16.783086    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" podUID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" containerName="maas-api" containerID="cri-o://26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44" gracePeriod=30
Nov 01 21:41:17.045121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.045061    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:41:17.221044 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221007    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\") pod \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") "
Nov 01 21:41:17.221044 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221048    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\" (UniqueName: \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\") pod \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") "
Nov 01 21:41:17.221252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221115    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\") pod \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") "
Nov 01 21:41:17.221252 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221217    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") pod \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\" (UID: \"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c\") "
Nov 01 21:41:17.221410 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221382    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca" pod "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" (UID: "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:41:17.221617 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.221593    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:17.223204 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.223173    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs" pod "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" (UID: "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:41:17.223314 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.223214    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q" pod "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" (UID: "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c"). InnerVolumeSpecName "kube-api-access-g978q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:17.223314 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.223250    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls" pod "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" (UID: "71386dbc-7b7f-4395-98f0-cf4fabe8fa5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:41:17.322263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.322184    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:17.322263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.322212    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g978q\" (UniqueName: \"kubernetes.io/projected/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-kube-api-access-g978q\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:17.322263 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.322222    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:17.747203 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.747167    5066 generic.go:356] "Generic (PLEG): container finished" podID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" containerID="26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44" exitCode=0
Nov 01 21:41:17.747398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.747215    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"
Nov 01 21:41:17.747398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.747250    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" event={"ID":"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c","Type":"ContainerDied","Data":"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"}
Nov 01 21:41:17.747398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.747301    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm" event={"ID":"71386dbc-7b7f-4395-98f0-cf4fabe8fa5c","Type":"ContainerDied","Data":"347f68fd6d5c006cdb3e3418dc1821cfd3024ab4249100364dfa4debfe86d100"}
Nov 01 21:41:17.747398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.747324    5066 scope.go:122] "RemoveContainer" containerID="26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"
Nov 01 21:41:17.757198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.757182    5066 scope.go:122] "RemoveContainer" containerID="26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"
Nov 01 21:41:17.757441 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:17.757425    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44\": container with ID starting with 26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44 not found: ID does not exist" containerID="26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"
Nov 01 21:41:17.757514 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.757448    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44"} err="failed to get container status \"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44\": rpc error: code = NotFound desc = could not find container \"26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44\": container with ID starting with 26066f60b80ab5f660ca943996e7e5b15ccf5fc7b4753fdde564f18957f12e44 not found: ID does not exist"
Nov 01 21:41:17.913827 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.913796    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:41:17.964299 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:17.964271    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c660c6-5b7c75f5cd-s2mlm"]
Nov 01 21:41:18.731188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:18.731154    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" path="/var/lib/kubelet/pods/71386dbc-7b7f-4395-98f0-cf4fabe8fa5c/volumes"
Nov 01 21:41:19.113254 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.113168    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" podUID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" containerName="payload-processing" containerID="cri-o://1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332" gracePeriod=25
Nov 01 21:41:19.207742 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.207711    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:41:19.207978 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.207957    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy" containerID="cri-o://f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab" gracePeriod=30
Nov 01 21:41:19.221313 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.221280    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" podUID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" containerName="payload-pre-processing" containerID="cri-o://1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2" gracePeriod=25
Nov 01 21:41:19.407029 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.406994    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:41:19.485385 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.485363    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:41:19.546226 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.546195    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\" (UniqueName: \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\") pod \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") "
Nov 01 21:41:19.546387 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.546368    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\") pod \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\" (UID: \"2566f94d-8b18-4a0a-bb5a-45d7335af9f1\") "
Nov 01 21:41:19.546738 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.546715    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume" pod "2566f94d-8b18-4a0a-bb5a-45d7335af9f1" (UID: "2566f94d-8b18-4a0a-bb5a-45d7335af9f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:41:19.546848 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.546826    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:19.548303 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.548277    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6" pod "2566f94d-8b18-4a0a-bb5a-45d7335af9f1" (UID: "2566f94d-8b18-4a0a-bb5a-45d7335af9f1"). InnerVolumeSpecName "kube-api-access-ts2w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:19.648202 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.648134    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\") pod \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") "
Nov 01 21:41:19.648324 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.648227    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\" (UniqueName: \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\") pod \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\" (UID: \"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33\") "
Nov 01 21:41:19.648589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.648564    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume" pod "1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" (UID: "1644ebd9-2ccb-4f93-acc1-1b5c6f647d33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:41:19.648719 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.648700    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts2w6\" (UniqueName: \"kubernetes.io/projected/2566f94d-8b18-4a0a-bb5a-45d7335af9f1-kube-api-access-ts2w6\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:19.648797 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.648724    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:19.650302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.650276    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w" pod "1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" (UID: "1644ebd9-2ccb-4f93-acc1-1b5c6f647d33"). InnerVolumeSpecName "kube-api-access-mvd8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:19.749720 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.749695    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvd8w\" (UniqueName: \"kubernetes.io/projected/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33-kube-api-access-mvd8w\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:19.759619 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.759592    5066 generic.go:356] "Generic (PLEG): container finished" podID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" containerID="1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2" exitCode=0
Nov 01 21:41:19.759750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.759637    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"
Nov 01 21:41:19.759750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.759675    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" event={"ID":"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33","Type":"ContainerDied","Data":"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"}
Nov 01 21:41:19.759750 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.759729    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7" event={"ID":"1644ebd9-2ccb-4f93-acc1-1b5c6f647d33","Type":"ContainerDied","Data":"2fb3ae74120179fbf0b26e3a58fdea04a4413696a119e3bfe70c84ab4a62f097"}
Nov 01 21:41:19.759916 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.759757    5066 scope.go:122] "RemoveContainer" containerID="1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"
Nov 01 21:41:19.761022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.760998    5066 generic.go:356] "Generic (PLEG): container finished" podID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" containerID="1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332" exitCode=0
Nov 01 21:41:19.761120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.761032    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"
Nov 01 21:41:19.761120 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.761083    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" event={"ID":"2566f94d-8b18-4a0a-bb5a-45d7335af9f1","Type":"ContainerDied","Data":"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"}
Nov 01 21:41:19.761234 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.761132    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv" event={"ID":"2566f94d-8b18-4a0a-bb5a-45d7335af9f1","Type":"ContainerDied","Data":"70d20c1ba6fbca360056fa304265103c227950a1c1e09580690081afe81dff6d"}
Nov 01 21:41:19.769068 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.769048    5066 scope.go:122] "RemoveContainer" containerID="1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"
Nov 01 21:41:19.769319 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:19.769302    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2\": container with ID starting with 1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2 not found: ID does not exist" containerID="1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"
Nov 01 21:41:19.769397 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.769333    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2"} err="failed to get container status \"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2\": rpc error: code = NotFound desc = could not find container \"1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2\": container with ID starting with 1676ad41db8f2ac0ae121937a6d8d6241fa2867cea67ae76adb73e3cadc536f2 not found: ID does not exist"
Nov 01 21:41:19.769397 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.769354    5066 scope.go:122] "RemoveContainer" containerID="1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"
Nov 01 21:41:19.777384 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.777365    5066 scope.go:122] "RemoveContainer" containerID="1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"
Nov 01 21:41:19.777633 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:19.777612    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332\": container with ID starting with 1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332 not found: ID does not exist" containerID="1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"
Nov 01 21:41:19.777715 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.777638    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332"} err="failed to get container status \"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332\": rpc error: code = NotFound desc = could not find container \"1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332\": container with ID starting with 1469cb7667dc12cb9d71e702dc39d4ad6ddd1e4453acc1b590925f2f11034332 not found: ID does not exist"
Nov 01 21:41:19.926542 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.926438    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:41:19.930116 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:19.930092    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c660c6-85ffdb5ffdtxbv"]
Nov 01 21:41:20.039422 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:20.039377    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:41:20.045514 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:20.045475    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c660c6-ddb9fpbhg7"]
Nov 01 21:41:20.730671 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:20.730641    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" path="/var/lib/kubelet/pods/1644ebd9-2ccb-4f93-acc1-1b5c6f647d33/volumes"
Nov 01 21:41:20.731043 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:20.731002    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" path="/var/lib/kubelet/pods/2566f94d-8b18-4a0a-bb5a-45d7335af9f1/volumes"
Nov 01 21:41:21.628426 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:21.628388    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:41:24.461663 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.461639    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:41:24.595034 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.594967    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595034 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595011    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595059    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595083    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595113    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595155    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595262    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595287    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595331    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\") pod \"832410b1-6377-434a-91db-971698485dd4\" (UID: \"832410b1-6377-434a-91db-971698485dd4\") "
Nov 01 21:41:24.595484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595399    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:24.595722 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595636    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:24.595778 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595757    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.595833 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595778    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.595888 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595832    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:24.595888 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595858    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:24.596000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.595933    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:41:24.597532 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.597485    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:41:24.598009 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.597985    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "kube-api-access-n446w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:24.598117 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.597999    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:24.598117 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.598030    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token" pod "832410b1-6377-434a-91db-971698485dd4" (UID: "832410b1-6377-434a-91db-971698485dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:24.696302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696275    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696298    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696308    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n446w\" (UniqueName: \"kubernetes.io/projected/832410b1-6377-434a-91db-971698485dd4-kube-api-access-n446w\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696317    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832410b1-6377-434a-91db-971698485dd4-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696326    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832410b1-6377-434a-91db-971698485dd4-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696336    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.696453 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.696345    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832410b1-6377-434a-91db-971698485dd4-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:24.792843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.792816    5066 generic.go:356] "Generic (PLEG): container finished" podID="832410b1-6377-434a-91db-971698485dd4" containerID="f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab" exitCode=0
Nov 01 21:41:24.792949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.792863    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"
Nov 01 21:41:24.792949 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.792900    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerDied","Data":"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"}
Nov 01 21:41:24.793041 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.792948    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx" event={"ID":"832410b1-6377-434a-91db-971698485dd4","Type":"ContainerDied","Data":"a4441045da983403f9b44144a234cd6c6bca4d765e962dd9766c30f54d543c9a"}
Nov 01 21:41:24.793041 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.792970    5066 scope.go:122] "RemoveContainer" containerID="f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"
Nov 01 21:41:24.802402 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.802373    5066 scope.go:122] "RemoveContainer" containerID="427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"
Nov 01 21:41:24.813114 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.813093    5066 scope.go:122] "RemoveContainer" containerID="f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"
Nov 01 21:41:24.813385 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:24.813367    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab\": container with ID starting with f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab not found: ID does not exist" containerID="f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"
Nov 01 21:41:24.813435 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.813391    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab"} err="failed to get container status \"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab\": rpc error: code = NotFound desc = could not find container \"f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab\": container with ID starting with f315db510f89e5c6902f6db2f0a35b696e2102f6fcff4fdca1d2458233465dab not found: ID does not exist"
Nov 01 21:41:24.813435 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.813412    5066 scope.go:122] "RemoveContainer" containerID="427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"
Nov 01 21:41:24.813696 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:24.813678    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341\": container with ID starting with 427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341 not found: ID does not exist" containerID="427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"
Nov 01 21:41:24.813754 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.813700    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341"} err="failed to get container status \"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341\": rpc error: code = NotFound desc = could not find container \"427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341\": container with ID starting with 427eb8147bd88681fef5c28c98f65b48e76711b86ec6ebd684a3afaafb221341 not found: ID does not exist"
Nov 01 21:41:24.962336 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.962260    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:41:24.967098 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:24.967077    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c660c6-openshift-default-bbfc647df-gqsmx"]
Nov 01 21:41:26.731568 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:26.731540    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832410b1-6377-434a-91db-971698485dd4" path="/var/lib/kubelet/pods/832410b1-6377-434a-91db-971698485dd4/volumes"
Nov 01 21:41:39.494256 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:39.494222    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 21:41:39.494775 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:39.494448    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" podUID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" containerName="manager" containerID="cri-o://f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7" gracePeriod=10
Nov 01 21:41:41.150384 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.150355    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 21:41:41.224433 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.224406    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\") pod \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") "
Nov 01 21:41:41.224575 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.224470    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\" (UniqueName: \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\") pod \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\" (UID: \"0da5efcb-cdd3-48d5-813d-17bc1d59717c\") "
Nov 01 21:41:41.224761 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.224738    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume" pod "0da5efcb-cdd3-48d5-813d-17bc1d59717c" (UID: "0da5efcb-cdd3-48d5-813d-17bc1d59717c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:41:41.226449 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.226428    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh" pod "0da5efcb-cdd3-48d5-813d-17bc1d59717c" (UID: "0da5efcb-cdd3-48d5-813d-17bc1d59717c"). InnerVolumeSpecName "kube-api-access-v9pvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:41:41.325970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.325944    5066 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0da5efcb-cdd3-48d5-813d-17bc1d59717c-extensions-socket-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:41.325970 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.325968    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9pvh\" (UniqueName: \"kubernetes.io/projected/0da5efcb-cdd3-48d5-813d-17bc1d59717c-kube-api-access-v9pvh\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:41:41.907348 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.907316    5066 generic.go:356] "Generic (PLEG): container finished" podID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" containerID="f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7" exitCode=0
Nov 01 21:41:41.907545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.907354    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"
Nov 01 21:41:41.907545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.907354    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" event={"ID":"0da5efcb-cdd3-48d5-813d-17bc1d59717c","Type":"ContainerDied","Data":"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"}
Nov 01 21:41:41.907545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.907473    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s" event={"ID":"0da5efcb-cdd3-48d5-813d-17bc1d59717c","Type":"ContainerDied","Data":"236fe54ceaa79b3f9bb446317d83cc1feab6c71e36d9233745b17d39c99a72cb"}
Nov 01 21:41:41.907545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.907520    5066 scope.go:122] "RemoveContainer" containerID="f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"
Nov 01 21:41:41.923405 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.923386    5066 scope.go:122] "RemoveContainer" containerID="f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"
Nov 01 21:41:41.923676 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:41.923660    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7\": container with ID starting with f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7 not found: ID does not exist" containerID="f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"
Nov 01 21:41:41.923722 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:41.923682    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7"} err="failed to get container status \"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7\": rpc error: code = NotFound desc = could not find container \"f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7\": container with ID starting with f5960f004516ec4473274e60af824a45d847820e420753168b67b1e8566144e7 not found: ID does not exist"
Nov 01 21:41:42.124626 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:42.124601    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 21:41:42.175428 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:42.175374    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mjh7s"]
Nov 01 21:41:42.729667 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:41:42.729632    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" path="/var/lib/kubelet/pods/0da5efcb-cdd3-48d5-813d-17bc1d59717c/volumes"
Nov 01 21:41:51.627816 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:41:51.627775    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:42:21.627721 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:42:21.627646    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:42:27.625152 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:27.625124    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:42:27.693298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:27.693272    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:42:42.988969 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.988935    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"]
Nov 01 21:42:42.989380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989349    5066 cpu_manager.go:414] "containerMap: removing container" podUID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" containerName="manager"
Nov 01 21:42:42.989380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989360    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" containerName="manager"
Nov 01 21:42:42.989380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989371    5066 cpu_manager.go:414] "containerMap: removing container" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989376    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989384    5066 cpu_manager.go:414] "containerMap: removing container" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989389    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989403    5066 cpu_manager.go:414] "containerMap: removing container" podUID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" containerName="maas-api"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989407    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" containerName="maas-api"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989417    5066 cpu_manager.go:414] "containerMap: removing container" podUID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" containerName="payload-pre-processing"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989422    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" containerName="payload-pre-processing"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989445    5066 cpu_manager.go:414] "containerMap: removing container" podUID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" containerName="payload-processing"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989449    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" containerName="payload-processing"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989534    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da5efcb-cdd3-48d5-813d-17bc1d59717c" containerName="manager"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989542    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989547    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="832410b1-6377-434a-91db-971698485dd4" containerName="istio-proxy"
Nov 01 21:42:42.989895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989555    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="71386dbc-7b7f-4395-98f0-cf4fabe8fa5c" containerName="maas-api"
Nov 01 21:42:42.989895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989560    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="1644ebd9-2ccb-4f93-acc1-1b5c6f647d33" containerName="payload-pre-processing"
Nov 01 21:42:42.989895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.989570    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="2566f94d-8b18-4a0a-bb5a-45d7335af9f1" containerName="payload-processing"
Nov 01 21:42:42.993899 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:42.993881    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.034530 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.034485    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"]
Nov 01 21:42:43.081447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.081412    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4n5t\""
Nov 01 21:42:43.117213 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.117181    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbg2\" (UniqueName: \"kubernetes.io/projected/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-kube-api-access-6wbg2\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.117366 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.117222    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.217901 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.217854    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbg2\" (UniqueName: \"kubernetes.io/projected/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-kube-api-access-6wbg2\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.217901 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.217905    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.218297 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.218277    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.275521 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.275436    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbg2\" (UniqueName: \"kubernetes.io/projected/6b6e656f-8138-4f53-835d-f3fa7b8bc9d2-kube-api-access-6wbg2\") pod \"kuadrant-operator-controller-manager-678d87766f-mfmm4\" (UID: \"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.306174 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.306140    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:43.780585 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:43.780552    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"]
Nov 01 21:42:43.782132 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:42:43.782102    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e656f_8138_4f53_835d_f3fa7b8bc9d2.slice/crio-a4046def2d99fbab546b18052cae64e0407123589366cb463274ada0e4733efd WatchSource:0}: Error finding container a4046def2d99fbab546b18052cae64e0407123589366cb463274ada0e4733efd: Status 404 returned error can't find the container with id a4046def2d99fbab546b18052cae64e0407123589366cb463274ada0e4733efd
Nov 01 21:42:44.225121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:44.225075    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4" event={"ID":"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2","Type":"ContainerStarted","Data":"ebd964a3082b7bebe29d0108143b334b877d57eddaf6bfad7d890804f23f9ca7"}
Nov 01 21:42:44.225526 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:44.225139    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:42:44.225526 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:44.225155    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4" event={"ID":"6b6e656f-8138-4f53-835d-f3fa7b8bc9d2","Type":"ContainerStarted","Data":"a4046def2d99fbab546b18052cae64e0407123589366cb463274ada0e4733efd"}
Nov 01 21:42:44.403128 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:44.403073    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4" podStartSLOduration=2.403059156 podStartE2EDuration="2.403059156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:42:44.399083908 +0000 UTC m=+6017.783981568" watchObservedRunningTime="2026-11-01 21:42:44.403059156 +0000 UTC m=+6017.787956746"
Nov 01 21:42:51.627295 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:42:51.627252    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:42:55.232381 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:42:55.232350    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mfmm4"
Nov 01 21:43:21.634281 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:43:21.634231    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:43:44.890432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.890340    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:44.894469 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.894444    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:44.900959 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.900932    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"]
Nov 01 21:43:44.904761 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.904741    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:44.947245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.947212    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:43:44.947245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.947227    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:43:44.950747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.950718    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:44.995571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.995538    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\""
Nov 01 21:43:44.995571 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.995537    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 21:43:44.995777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.995669    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:43:44.999562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:44.999541    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"]
Nov 01 21:43:45.015302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015251    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e693c822-8f7c-4b73-b122-dba7095bca4c-openshift-service-ca\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.015302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015308    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-maas-api-tls\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.015516 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015466    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.015596 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015576    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.015635 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015616    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdgc\" (UniqueName: \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.015694 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015680    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-metrics-certs\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.015770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015753    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.015816 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.015800    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2pl\" (UniqueName: \"kubernetes.io/projected/e693c822-8f7c-4b73-b122-dba7095bca4c-kube-api-access-dm2pl\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.117063 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117025    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.117063 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117065    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdgc\" (UniqueName: \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117118    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-metrics-certs\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117171    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117220    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2pl\" (UniqueName: \"kubernetes.io/projected/e693c822-8f7c-4b73-b122-dba7095bca4c-kube-api-access-dm2pl\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117246    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e693c822-8f7c-4b73-b122-dba7095bca4c-openshift-service-ca\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117280    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-maas-api-tls\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.117342 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.117334    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.118048 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.118018    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.118198 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.118124    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e693c822-8f7c-4b73-b122-dba7095bca4c-openshift-service-ca\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.120000 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.119970    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-maas-api-tls\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.120107 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.120037    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.120206 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.120190    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.120245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.120222    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e693c822-8f7c-4b73-b122-dba7095bca4c-metrics-certs\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.175465 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.175385    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2pl\" (UniqueName: \"kubernetes.io/projected/e693c822-8f7c-4b73-b122-dba7095bca4c-kube-api-access-dm2pl\") pod \"maas-api-8595f64f4d-kz76k\" (UID: \"e693c822-8f7c-4b73-b122-dba7095bca4c\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.175465 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.175415    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdgc\" (UniqueName: \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\") pod \"maas-api-845db7b4d-vbl4n\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.208412 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.208377    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:45.218005 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.217977    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:45.598374 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.598347    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:45.599393 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:43:45.599364    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92d3936_6a99_4c3c_bb6f_6a8bdbdcfa0a.slice/crio-490f05b7dabc4ab97a43c1094e9ac45479e4a2a2c953011b1dff1265ef4e4df6 WatchSource:0}: Error finding container 490f05b7dabc4ab97a43c1094e9ac45479e4a2a2c953011b1dff1265ef4e4df6: Status 404 returned error can't find the container with id 490f05b7dabc4ab97a43c1094e9ac45479e4a2a2c953011b1dff1265ef4e4df6
Nov 01 21:43:45.638329 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.638292    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"]
Nov 01 21:43:45.642899 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:43:45.642872    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode693c822_8f7c_4b73_b122_dba7095bca4c.slice/crio-2c8e33fff1fec9b09711f7deddd4b82f242a3b717e8d1a1fc4b48cc949d5279a WatchSource:0}: Error finding container 2c8e33fff1fec9b09711f7deddd4b82f242a3b717e8d1a1fc4b48cc949d5279a: Status 404 returned error can't find the container with id 2c8e33fff1fec9b09711f7deddd4b82f242a3b717e8d1a1fc4b48cc949d5279a
Nov 01 21:43:45.710780 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.710743    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k" event={"ID":"e693c822-8f7c-4b73-b122-dba7095bca4c","Type":"ContainerStarted","Data":"2c8e33fff1fec9b09711f7deddd4b82f242a3b717e8d1a1fc4b48cc949d5279a"}
Nov 01 21:43:45.712021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:45.711995    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" event={"ID":"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a","Type":"ContainerStarted","Data":"490f05b7dabc4ab97a43c1094e9ac45479e4a2a2c953011b1dff1265ef4e4df6"}
Nov 01 21:43:46.718897 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.718859    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k" event={"ID":"e693c822-8f7c-4b73-b122-dba7095bca4c","Type":"ContainerStarted","Data":"e97a7f9102563491d8530f29d14f490a6a5fbe470490de259de3528218cfd9eb"}
Nov 01 21:43:46.719363 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.719054    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:46.720615 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.720591    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" event={"ID":"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a","Type":"ContainerStarted","Data":"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"}
Nov 01 21:43:46.720747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.720724    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:46.837371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.837271    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k" podStartSLOduration=1.9528224440000002 podStartE2EDuration="2.837256209s" totalImagesPullingTime="884.433765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:46.833598455 +0000 UTC m=+6080.218496048" watchObservedRunningTime="2026-11-01 21:43:46.837256209 +0000 UTC m=+6080.222153801"
Nov 01 21:43:46.953642 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:46.953590    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" podStartSLOduration=2.064196877 podStartE2EDuration="2.953576528s" totalImagesPullingTime="889.379651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:46.951564613 +0000 UTC m=+6080.336462205" watchObservedRunningTime="2026-11-01 21:43:46.953576528 +0000 UTC m=+6080.338474119"
Nov 01 21:43:51.628039 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:43:51.628000    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:43:52.732380 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:52.732352    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:52.732797 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:52.732410    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-kz76k"
Nov 01 21:43:52.905291 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:52.905254    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:52.905561 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:52.905450    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" podUID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" containerName="maas-api" containerID="cri-o://f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928" gracePeriod=30
Nov 01 21:43:53.172408 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.172381    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:53.305478 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.305399    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\") pod \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") "
Nov 01 21:43:53.305675 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.305521    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\") pod \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") "
Nov 01 21:43:53.305675 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.305557    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\" (UniqueName: \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\") pod \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") "
Nov 01 21:43:53.305675 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.305592    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\") pod \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\" (UID: \"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a\") "
Nov 01 21:43:53.305929 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.305827    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca" pod "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" (UID: "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:43:53.306080 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.306060    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:43:53.308193 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.308159    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs" pod "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" (UID: "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:43:53.308333 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.308259    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls" pod "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" (UID: "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:43:53.308913 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.308885    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc" pod "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" (UID: "f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a"). InnerVolumeSpecName "kube-api-access-jsdgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:43:53.407676 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.407642    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:43:53.407676 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.407675    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdgc\" (UniqueName: \"kubernetes.io/projected/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-kube-api-access-jsdgc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:43:53.407885 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.407687    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:43:53.757060 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.757027    5066 generic.go:356] "Generic (PLEG): container finished" podID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" containerID="f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928" exitCode=0
Nov 01 21:43:53.757562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.757089    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"
Nov 01 21:43:53.757562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.757117    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" event={"ID":"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a","Type":"ContainerDied","Data":"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"}
Nov 01 21:43:53.757562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.757183    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n" event={"ID":"f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a","Type":"ContainerDied","Data":"490f05b7dabc4ab97a43c1094e9ac45479e4a2a2c953011b1dff1265ef4e4df6"}
Nov 01 21:43:53.757562 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.757219    5066 scope.go:122] "RemoveContainer" containerID="f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"
Nov 01 21:43:53.769824 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.769806    5066 scope.go:122] "RemoveContainer" containerID="f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"
Nov 01 21:43:53.770072 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:43:53.770051    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928\": container with ID starting with f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928 not found: ID does not exist" containerID="f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"
Nov 01 21:43:53.770148 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.770084    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928"} err="failed to get container status \"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928\": rpc error: code = NotFound desc = could not find container \"f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928\": container with ID starting with f14d1d7dfa607df66295e8ed888801800ed06b13da93743c0173999b5d413928 not found: ID does not exist"
Nov 01 21:43:53.923917 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.923879    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:53.927223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:53.927201    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-vbl4n"]
Nov 01 21:43:54.730484 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:43:54.730447    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" path="/var/lib/kubelet/pods/f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a/volumes"
Nov 01 21:44:21.628862 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:44:21.628827    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:44:51.628056 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:44:51.628013    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:45:21.629431 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:45:21.629344    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:45:51.627133 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:45:51.627097    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:46:21.629025 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:46:21.628980    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:46:51.628922 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:46:51.628838    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:47:21.628385 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:47:21.628350    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:47:27.661869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:47:27.661841    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:47:27.731325 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:47:27.731298    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:47:51.628471 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:47:51.628432    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:48:21.628909 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:48:21.628832    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:48:51.628208 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:48:51.628170    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:49:21.628611 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:49:21.628571    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:49:51.628663 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:49:51.628579    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:50:21.629635 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:50:21.629586    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:50:51.628778 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:50:51.628737    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:51:21.629509 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:51:21.629415    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:51:51.628452 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:51:51.628411    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:52:21.629803 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:21.629762    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:52:27.698860 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:27.698829    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:52:27.772883 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:27.772853    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:52:51.383910 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.383813    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:52:51.384402 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.384086    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy" containerID="cri-o://cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a" gracePeriod=30
Nov 01 21:52:51.434441 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.434399    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:52:51.434669 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.434620    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy" containerID="cri-o://f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a" gracePeriod=30
Nov 01 21:52:51.437863 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.437833    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:52:51.438138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.438046    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy" containerID="cri-o://1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f" gracePeriod=30
Nov 01 21:52:51.485019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.484986    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:52:51.485207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:51.485179    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy" containerID="cri-o://23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b" gracePeriod=30
Nov 01 21:52:51.630845 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:51.630805    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:52:56.702532 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.702376    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:52:56.739803 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.739768    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.739890    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.739913    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.739944    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.739985    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740016    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740105    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740102    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740136    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740162    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\") pod \"69fab25f-0b64-4191-8278-79316172a3b0\" (UID: \"69fab25f-0b64-4191-8278-79316172a3b0\") "
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740352    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740459    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.740511 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740472    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.741296 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.740927    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:52:56.741296 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.741227    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.741473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.741428    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.743826 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.743793    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:52:56.743826 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.743809    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.744008 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.743934    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.746664 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.746638    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh" pod "69fab25f-0b64-4191-8278-79316172a3b0" (UID: "69fab25f-0b64-4191-8278-79316172a3b0"). InnerVolumeSpecName "kube-api-access-kj2hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.792867 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.792840    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:52:56.802986 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.802959    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:52:56.815821 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.815796    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:52:56.841444 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841412    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841457    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841475    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841513    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841658 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841625    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841681    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.841907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841741    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.841907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841776    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.841907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841825    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.841907 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841897    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841936    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.841965    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842001    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\") pod \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\" (UID: \"33adf5c9-ce85-4cea-abdd-ae9834ebd472\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842032    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842066    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842088    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842097    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842142    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842188 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842189    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\") pod \"bfc0490c-6130-47ff-ad1d-36a80b14741b\" (UID: \"bfc0490c-6130-47ff-ad1d-36a80b14741b\") "
Nov 01 21:52:56.842780 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842753    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fab25f-0b64-4191-8278-79316172a3b0-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.842857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842782    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2hh\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-kube-api-access-kj2hh\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.842857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842800    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.842857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842818    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fab25f-0b64-4191-8278-79316172a3b0-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.842857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842832    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.842857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842847    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.843219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842863    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fab25f-0b64-4191-8278-79316172a3b0-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.843219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842878    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fab25f-0b64-4191-8278-79316172a3b0-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.845203 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842092    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845203 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842287    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842299    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.842690    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.843655    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.844318    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.844800    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.844834    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.844982    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.845379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.845171    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:52:56.845843 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.845793    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:52:56.845982 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.845953    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.846114 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.845993    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.846237 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.846206    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j" pod "33adf5c9-ce85-4cea-abdd-ae9834ebd472" (UID: "33adf5c9-ce85-4cea-abdd-ae9834ebd472"). InnerVolumeSpecName "kube-api-access-5pt2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.846350 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.846282    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.846818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.846796    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:52:56.847260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.847229    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr" pod "bfc0490c-6130-47ff-ad1d-36a80b14741b" (UID: "bfc0490c-6130-47ff-ad1d-36a80b14741b"). InnerVolumeSpecName "kube-api-access-8vvtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.880895 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.880860    5066 generic.go:356] "Generic (PLEG): container finished" podID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerID="1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f" exitCode=0
Nov 01 21:52:56.881090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.880907    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"
Nov 01 21:52:56.881090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.880947    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" event={"ID":"bfc0490c-6130-47ff-ad1d-36a80b14741b","Type":"ContainerDied","Data":"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"}
Nov 01 21:52:56.881090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.881011    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf" event={"ID":"bfc0490c-6130-47ff-ad1d-36a80b14741b","Type":"ContainerDied","Data":"f9c18577d188c7f78365215a170eda976ab124ed888d369c56bc6c4e380962f4"}
Nov 01 21:52:56.881090 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.881043    5066 scope.go:122] "RemoveContainer" containerID="1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"
Nov 01 21:52:56.882873 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.882776    5066 generic.go:356] "Generic (PLEG): container finished" podID="69fab25f-0b64-4191-8278-79316172a3b0" containerID="cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a" exitCode=0
Nov 01 21:52:56.882873 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.882825    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"
Nov 01 21:52:56.882873 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.882857    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" event={"ID":"69fab25f-0b64-4191-8278-79316172a3b0","Type":"ContainerDied","Data":"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"}
Nov 01 21:52:56.883144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.882907    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7" event={"ID":"69fab25f-0b64-4191-8278-79316172a3b0","Type":"ContainerDied","Data":"dd033650695d839c7c648966912498e94ee491d26c7576956c1d0cb16675ea49"}
Nov 01 21:52:56.884756 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.884733    5066 generic.go:356] "Generic (PLEG): container finished" podID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerID="23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b" exitCode=0
Nov 01 21:52:56.884887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.884790    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"
Nov 01 21:52:56.884887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.884820    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" event={"ID":"e52ff78c-5fad-4952-93b4-c5a6d031159e","Type":"ContainerDied","Data":"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"}
Nov 01 21:52:56.884887 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.884869    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7" event={"ID":"e52ff78c-5fad-4952-93b4-c5a6d031159e","Type":"ContainerDied","Data":"6b330f016071e314c0ca39724b3021c3e3ba312e293e3bfd4406cfdb11260b1d"}
Nov 01 21:52:56.886601 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.886577    5066 generic.go:356] "Generic (PLEG): container finished" podID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerID="f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a" exitCode=0
Nov 01 21:52:56.886732 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.886634    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"
Nov 01 21:52:56.886732 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.886657    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" event={"ID":"33adf5c9-ce85-4cea-abdd-ae9834ebd472","Type":"ContainerDied","Data":"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"}
Nov 01 21:52:56.886732 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.886699    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p" event={"ID":"33adf5c9-ce85-4cea-abdd-ae9834ebd472","Type":"ContainerDied","Data":"df1458550e15771a35fbeae5c857d8f0882b49e6216c00aa36a21524a1e5fe70"}
Nov 01 21:52:56.896776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.896641    5066 scope.go:122] "RemoveContainer" containerID="1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"
Nov 01 21:52:56.897057 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:56.897030    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f\": container with ID starting with 1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f not found: ID does not exist" containerID="1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"
Nov 01 21:52:56.897139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.897069    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f"} err="failed to get container status \"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f\": rpc error: code = NotFound desc = could not find container \"1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f\": container with ID starting with 1b8a01d2808b9920ff50528a3b7a9857eb28b81f7e65342c0407756f812fed4f not found: ID does not exist"
Nov 01 21:52:56.897139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.897100    5066 scope.go:122] "RemoveContainer" containerID="cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"
Nov 01 21:52:56.907963 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.907876    5066 scope.go:122] "RemoveContainer" containerID="cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"
Nov 01 21:52:56.909426 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:56.909400    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a\": container with ID starting with cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a not found: ID does not exist" containerID="cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"
Nov 01 21:52:56.909565 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.909433    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a"} err="failed to get container status \"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a\": rpc error: code = NotFound desc = could not find container \"cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a\": container with ID starting with cda57b60af9c98a4a891a31179233439702fc472acc0b5ef9309e05e61ce215a not found: ID does not exist"
Nov 01 21:52:56.909565 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.909452    5066 scope.go:122] "RemoveContainer" containerID="23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"
Nov 01 21:52:56.918725 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.918696    5066 scope.go:122] "RemoveContainer" containerID="23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"
Nov 01 21:52:56.919003 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:56.918982    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b\": container with ID starting with 23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b not found: ID does not exist" containerID="23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"
Nov 01 21:52:56.919073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.919012    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b"} err="failed to get container status \"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b\": rpc error: code = NotFound desc = could not find container \"23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b\": container with ID starting with 23ba1a4154b61c781bf4953eea4fcfff14847dd9dded19f00ff441170d76580b not found: ID does not exist"
Nov 01 21:52:56.919073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.919030    5066 scope.go:122] "RemoveContainer" containerID="f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"
Nov 01 21:52:56.927434 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.927418    5066 scope.go:122] "RemoveContainer" containerID="f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"
Nov 01 21:52:56.927700 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:52:56.927679    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a\": container with ID starting with f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a not found: ID does not exist" containerID="f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"
Nov 01 21:52:56.927760 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.927707    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a"} err="failed to get container status \"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a\": rpc error: code = NotFound desc = could not find container \"f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a\": container with ID starting with f372c008be2b9336e07d21c4d6eb75d218a963055469e6e433da95bc1524820a not found: ID does not exist"
Nov 01 21:52:56.947761 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947734    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.947904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947772    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.947904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947806    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.947904 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947855    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947924    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.947951    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948048    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948116    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948156    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\") pod \"e52ff78c-5fad-4952-93b4-c5a6d031159e\" (UID: \"e52ff78c-5fad-4952-93b4-c5a6d031159e\") "
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948307    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948539    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948561    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948578    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52ff78c-5fad-4952-93b4-c5a6d031159e-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.948593 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948593    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948610    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948624    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pt2j\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-kube-api-access-5pt2j\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948640    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948655    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948670    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948685    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc0490c-6130-47ff-ad1d-36a80b14741b-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948700    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948714    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvtr\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-kube-api-access-8vvtr\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948727    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc0490c-6130-47ff-ad1d-36a80b14741b-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948740    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc0490c-6130-47ff-ad1d-36a80b14741b-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948753    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948766    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948782    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.949304 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.948795    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33adf5c9-ce85-4cea-abdd-ae9834ebd472-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:56.950616 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.949883    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.950616 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.949903    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.950616 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.949922    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.950616 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.949962    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.951026 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.950997    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:52:56.951127 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.951103    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:52:56.951216 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.951196    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "kube-api-access-j6bq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:56.952346 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:56.952322    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token" pod "e52ff78c-5fad-4952-93b4-c5a6d031159e" (UID: "e52ff78c-5fad-4952-93b4-c5a6d031159e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:52:57.049473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049429    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049466    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049473 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049480    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049523    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049536    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049551    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52ff78c-5fad-4952-93b4-c5a6d031159e-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049562    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bq4\" (UniqueName: \"kubernetes.io/projected/e52ff78c-5fad-4952-93b4-c5a6d031159e-kube-api-access-j6bq4\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.049789 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.049574    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52ff78c-5fad-4952-93b4-c5a6d031159e-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:52:57.295283 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.295246    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:52:57.300637 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.300609    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fmf7p"]
Nov 01 21:52:57.411624 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.411586    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:52:57.418653 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.418618    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pgrt7"]
Nov 01 21:52:57.527747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.527709    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:52:57.530857 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.530830    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ljwmf"]
Nov 01 21:52:57.693284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.693245    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:52:57.696432 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:57.696407    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vdzv7"]
Nov 01 21:52:58.730360 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:58.730321    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" path="/var/lib/kubelet/pods/33adf5c9-ce85-4cea-abdd-ae9834ebd472/volumes"
Nov 01 21:52:58.730822 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:58.730807    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fab25f-0b64-4191-8278-79316172a3b0" path="/var/lib/kubelet/pods/69fab25f-0b64-4191-8278-79316172a3b0/volumes"
Nov 01 21:52:58.731203 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:58.731190    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" path="/var/lib/kubelet/pods/bfc0490c-6130-47ff-ad1d-36a80b14741b/volumes"
Nov 01 21:52:58.731883 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:52:58.731866    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" path="/var/lib/kubelet/pods/e52ff78c-5fad-4952-93b4-c5a6d031159e/volumes"
Nov 01 21:53:21.634246 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:53:21.634205    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:53:51.629005 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:53:51.628958    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:54:21.629379 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:54:21.629296    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:54:37.174966 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.174925    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"]
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175334    5066 cpu_manager.go:414] "containerMap: removing container" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175344    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175363    5066 cpu_manager.go:414] "containerMap: removing container" podUID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" containerName="maas-api"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175369    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" containerName="maas-api"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175379    5066 cpu_manager.go:414] "containerMap: removing container" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175384    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175394    5066 cpu_manager.go:414] "containerMap: removing container" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175398    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175410    5066 cpu_manager.go:414] "containerMap: removing container" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175415    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175488    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52ff78c-5fad-4952-93b4-c5a6d031159e" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175512    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fab25f-0b64-4191-8278-79316172a3b0" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175522    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="33adf5c9-ce85-4cea-abdd-ae9834ebd472" containerName="istio-proxy"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175530    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92d3936-6a99-4c3c-bb6f-6a8bdbdcfa0a" containerName="maas-api"
Nov 01 21:54:37.175805 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.175537    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc0490c-6130-47ff-ad1d-36a80b14741b" containerName="istio-proxy"
Nov 01 21:54:37.178770 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.178752    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.187576 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.187552    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"]
Nov 01 21:54:37.191166 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.191146    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.231687 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.231651    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\""
Nov 01 21:54:37.280182 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.280145    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"]
Nov 01 21:54:37.325793 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.325751    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ecb3f1-43e1-44dd-b904-055ce99077b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.326068 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326035    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.326223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326072    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9hv\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-kube-api-access-6s9hv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.326223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326158    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.326223 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326208    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326254    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326353    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326395    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326433    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326529    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.326688 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326577    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326719    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326757    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4gr\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-kube-api-access-xs4gr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326787    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326877    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326917    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326964    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.327055 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.326998    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.329810 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.329788    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"]
Nov 01 21:54:37.427696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427606    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.427696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427640    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9hv\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-kube-api-access-6s9hv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.427696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427660    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427768    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427823    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427870    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427916    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427953    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.427988    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428019 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428012    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428050    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428076    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4gr\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-kube-api-access-xs4gr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428107    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428139    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428171    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428229    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428253    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428264    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428317    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ecb3f1-43e1-44dd-b904-055ce99077b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428323    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.428388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428388    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.429042 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428536    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.429042 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428896    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.429042 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.428958    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.429273 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.429073    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ecb3f1-43e1-44dd-b904-055ce99077b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.429273 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.429135    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.429273 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.429177    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.429632 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.429611    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.430315 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.430296    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.430586 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.430566    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.430753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.430736    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.430888 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.430867    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.485923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.485884    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.486106 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.485884    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.486106 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.486035    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9hv\" (UniqueName: \"kubernetes.io/projected/19ecb3f1-43e1-44dd-b904-055ce99077b7-kube-api-access-6s9hv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pnwrg\" (UID: \"19ecb3f1-43e1-44dd-b904-055ce99077b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.486232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.486106    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4gr\" (UniqueName: \"kubernetes.io/projected/61a6346f-9976-47c6-9006-8fbfdc4f07e5-kube-api-access-xs4gr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qr2z2\" (UID: \"61a6346f-9976-47c6-9006-8fbfdc4f07e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.492458 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.492433    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:37.503682 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.503651    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:37.926767 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.926616    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"]
Nov 01 21:54:37.927668 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:54:37.927635    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ecb3f1_43e1_44dd_b904_055ce99077b7.slice/crio-2ef78664fec5dcd1946a70bb80e188e7017b7d49a0dc7a82dea4690539decdca WatchSource:0}: Error finding container 2ef78664fec5dcd1946a70bb80e188e7017b7d49a0dc7a82dea4690539decdca: Status 404 returned error can't find the container with id 2ef78664fec5dcd1946a70bb80e188e7017b7d49a0dc7a82dea4690539decdca
Nov 01 21:54:37.930486 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.930464    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"]
Nov 01 21:54:37.931049 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.931017    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:37.931168 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.931090    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:37.931168 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.931134    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:37.931452 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:54:37.931426    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a6346f_9976_47c6_9006_8fbfdc4f07e5.slice/crio-655a1881c7681f2867b4d7d6707e0d9bfe25802986f35c727f9ff40628483e0b WatchSource:0}: Error finding container 655a1881c7681f2867b4d7d6707e0d9bfe25802986f35c727f9ff40628483e0b: Status 404 returned error can't find the container with id 655a1881c7681f2867b4d7d6707e0d9bfe25802986f35c727f9ff40628483e0b
Nov 01 21:54:37.933757 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.933726    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:37.933824 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.933806    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:37.933871 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:37.933856    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:38.433517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.432953    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" event={"ID":"61a6346f-9976-47c6-9006-8fbfdc4f07e5","Type":"ContainerStarted","Data":"c80bbf4c19723481039b020feefcbfbce142cf44a8fb3727cae95e8eab62eee5"}
Nov 01 21:54:38.433517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.433034    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" event={"ID":"61a6346f-9976-47c6-9006-8fbfdc4f07e5","Type":"ContainerStarted","Data":"655a1881c7681f2867b4d7d6707e0d9bfe25802986f35c727f9ff40628483e0b"}
Nov 01 21:54:38.441520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.436077    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" event={"ID":"19ecb3f1-43e1-44dd-b904-055ce99077b7","Type":"ContainerStarted","Data":"b1cd2e8d4f488f6e85be88675c41cf08557bef340a234b729764743ea1c9b5c5"}
Nov 01 21:54:38.441520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.436141    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" event={"ID":"19ecb3f1-43e1-44dd-b904-055ce99077b7","Type":"ContainerStarted","Data":"2ef78664fec5dcd1946a70bb80e188e7017b7d49a0dc7a82dea4690539decdca"}
Nov 01 21:54:38.498517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.497588    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:38.510517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.506593    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:38.556490 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:38.556426    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podStartSLOduration=1.556411405 podStartE2EDuration="1.556411405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:38.553412383 +0000 UTC m=+6731.938309973" watchObservedRunningTime="2026-11-01 21:54:38.556411405 +0000 UTC m=+6731.941308998"
Nov 01 21:54:39.506518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:39.494606    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:39.506518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:39.494674    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:39.511540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:39.509924    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:39.511540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:39.509997    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:40.500518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:40.499927    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:40.500518 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:40.500003    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:40.511384 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:40.511347    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:40.512609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:40.512548    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:41.511346 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:41.511305    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:41.511604 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:41.511371    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:41.512018 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:41.511821    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:41.512018 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:41.511846    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:42.494959 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:42.494914    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:42.495269 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:42.495240    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:42.508428 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:42.508394    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:42.508732 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:42.508699    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:43.503520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:43.494202    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:43.503520 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:43.494266    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:43.506250 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:43.506218    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:43.506667 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:43.506637    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:44.493454 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:44.493414    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:44.493665 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:44.493478    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:44.504772 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:44.504735    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:44.505118 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:44.504792    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:45.493279 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:45.493243    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Nov 01 21:54:45.493447 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:45.493297    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Nov 01 21:54:45.504517 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:45.504477    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 01 21:54:45.504666 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:45.504540    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 01 21:54:47.491475 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.491437    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:47.492018 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.491993    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:47.492972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.492946    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2"
Nov 01 21:54:47.526548 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.526513    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:54:47.526692 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.526566    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:54:47.534388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.534363    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:47.534721 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.534701    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:47.535454 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.535432    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg"
Nov 01 21:54:47.702712 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:47.702648    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podStartSLOduration=10.702632832 podStartE2EDuration="10.702632832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:38.694227223 +0000 UTC m=+6732.079124817" watchObservedRunningTime="2026-11-01 21:54:47.702632832 +0000 UTC m=+6741.087530423"
Nov 01 21:54:51.628319 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:54:51.628279    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:54:52.312040 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.311997    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"]
Nov 01 21:54:52.315972 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.315943    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.411746 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.411708    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.411912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.411770    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.411912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.411878    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.411912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.411902    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.412059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.411946    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.412059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.412031    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.412158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.412067    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.412158 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.412119    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.412260 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.412181    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzksw\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-kube-api-access-bzksw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.512827 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512790    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.512827 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512833    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512893    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512919    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512942    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.512963    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.513000    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzksw\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-kube-api-access-bzksw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.513031    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.513086    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.513444    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.513545 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.513513    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.515799 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.515770    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.517696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.517646    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.517696 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.517684    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"]
Nov 01 21:54:52.518059 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.517996    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.518159 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.518097    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.524370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.524344    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.570845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.570782    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.570992 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.570867    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzksw\" (UniqueName: \"kubernetes.io/projected/e7e5f891-7a78-440a-ad1f-2eedf6aad509-kube-api-access-bzksw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-p6j5v\" (UID: \"e7e5f891-7a78-440a-ad1f-2eedf6aad509\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:52.628968 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:52.628928    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:53.059530 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.059484    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"]
Nov 01 21:54:53.059787 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:54:53.059761    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e5f891_7a78_440a_ad1f_2eedf6aad509.slice/crio-219992372c9ebfc4f8a643dd709b32dcfde243fa4663874453718a45510dcd6f WatchSource:0}: Error finding container 219992372c9ebfc4f8a643dd709b32dcfde243fa4663874453718a45510dcd6f: Status 404 returned error can't find the container with id 219992372c9ebfc4f8a643dd709b32dcfde243fa4663874453718a45510dcd6f
Nov 01 21:54:53.061927 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.061895    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:53.062023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.061957    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:53.062023 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.061984    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:53.539137 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.539102    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" event={"ID":"e7e5f891-7a78-440a-ad1f-2eedf6aad509","Type":"ContainerStarted","Data":"137e029fc304d7a4cd34389b0bfbab364d2ff068158eca658edb3e2a3d03d92e"}
Nov 01 21:54:53.539331 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.539160    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" event={"ID":"e7e5f891-7a78-440a-ad1f-2eedf6aad509","Type":"ContainerStarted","Data":"219992372c9ebfc4f8a643dd709b32dcfde243fa4663874453718a45510dcd6f"}
Nov 01 21:54:53.630009 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.629961    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:54:53.710162 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:53.710106    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podStartSLOduration=1.710092221 podStartE2EDuration="1.710092221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:53.706827254 +0000 UTC m=+6747.091724847" watchObservedRunningTime="2026-11-01 21:54:53.710092221 +0000 UTC m=+6747.094989812"
Nov 01 21:54:54.631573 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:54.631530    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:54.632011 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:54.631588    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:54:55.629425 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:55.629387    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:55.629650 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:55.629443    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:54:56.629637 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:56.629601    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:56.630139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:56.629675    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:54:57.629287 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:57.629251    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:57.629481 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:57.629315    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:54:58.630172 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:58.630132    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:58.630689 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:58.630192    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:54:59.630285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:59.630246    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:54:59.630741 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:54:59.630315    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:55:00.629708 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:55:00.629672    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 21:55:00.629930 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:55:00.629746    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 21:55:02.493370 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:55:02.493333    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:55:02.494173 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:55:02.494146    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:55:02.494958 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:55:02.494935    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v"
Nov 01 21:55:21.628977 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:55:21.628926    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:55:51.628697 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:55:51.628613    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:56:21.629236 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:56:21.629192    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:56:22.967950 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:22.967911    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"]
Nov 01 21:56:22.973261 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:22.973230    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.063844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.063812    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.063844 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.063847    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.063894    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2459e512-5a19-46b0-967d-881f764b0c8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064103    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjhc\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-kube-api-access-jrjhc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064151    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2459e512-5a19-46b0-967d-881f764b0c8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064193    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064246    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064768 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064426    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.064890 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.064807    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.081218 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.081179    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"]
Nov 01 21:56:23.165876 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.165833    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166071 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.165892    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166071 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.165919    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166071 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.165991    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2459e512-5a19-46b0-967d-881f764b0c8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166071 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166042    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjhc\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-kube-api-access-jrjhc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166077    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2459e512-5a19-46b0-967d-881f764b0c8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166125    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166186    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166294 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166229    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166554 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166326    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166554 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166354    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166591    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166684 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166671    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.166916 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.166891    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2459e512-5a19-46b0-967d-881f764b0c8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.168504 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.168470    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2459e512-5a19-46b0-967d-881f764b0c8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.168721 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.168702    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2459e512-5a19-46b0-967d-881f764b0c8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.224245 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.224172    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.224396 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.224241    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjhc\" (UniqueName: \"kubernetes.io/projected/2459e512-5a19-46b0-967d-881f764b0c8d-kube-api-access-jrjhc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7lgkh\" (UID: \"2459e512-5a19-46b0-967d-881f764b0c8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.286879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.286831    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:23.716662 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.716624    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"]
Nov 01 21:56:23.716923 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:56:23.716895    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2459e512_5a19_46b0_967d_881f764b0c8d.slice/crio-f4de002fddd7e9a3c83fcb415bdaf8a9c3054c5a8c608f0a08a852a292cee028 WatchSource:0}: Error finding container f4de002fddd7e9a3c83fcb415bdaf8a9c3054c5a8c608f0a08a852a292cee028: Status 404 returned error can't find the container with id f4de002fddd7e9a3c83fcb415bdaf8a9c3054c5a8c608f0a08a852a292cee028
Nov 01 21:56:23.719196 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.719162    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:23.719285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.719232    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:23.719285 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:23.719265    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:24.094040 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.094005    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" event={"ID":"2459e512-5a19-46b0-967d-881f764b0c8d","Type":"ContainerStarted","Data":"8fce27e91e29cf3ac61601fcca28b712fb1d67e9b324add888cda2667d46d5af"}
Nov 01 21:56:24.094456 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.094061    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" event={"ID":"2459e512-5a19-46b0-967d-881f764b0c8d","Type":"ContainerStarted","Data":"f4de002fddd7e9a3c83fcb415bdaf8a9c3054c5a8c608f0a08a852a292cee028"}
Nov 01 21:56:24.214445 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.214383    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podStartSLOduration=2.214367454 podStartE2EDuration="2.214367454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:24.211022883 +0000 UTC m=+6837.595920474" watchObservedRunningTime="2026-11-01 21:56:24.214367454 +0000 UTC m=+6837.599265045"
Nov 01 21:56:24.287912 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.287872    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:24.289437 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.289401    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:24.289609 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:24.289457    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:25.287983 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:25.287940    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:25.288415 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:25.288004    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:26.287765 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:26.287718    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:26.288012 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:26.287780    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:27.287351 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:27.287311    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:27.287589 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:27.287380    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:28.288107 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:28.288060    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:28.288697 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:28.288120    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:29.287801 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:29.287759    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:29.288028 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:29.287822    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:30.288208 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:30.288161    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:30.288663 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:30.288226    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:31.287416 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:31.287381    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 21:56:31.287608 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:31.287433    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 21:56:32.933677 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:32.933632    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:32.934354 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:32.934316    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:32.934975 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:56:32.934955    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh"
Nov 01 21:56:51.628198 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:56:51.628156    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:57:21.629468 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:57:21.629382    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:57:27.737295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:27.737264    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:57:27.812753 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:27.812721    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 21:57:37.665553 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.665516    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 21:57:37.669240 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.669221    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.734777 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.734740    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.734965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.734789    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvlw\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.734965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.734902    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.734965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.734929    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.734965 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.734959    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.735139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.735017    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.735139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.735061    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.735139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.735100    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.735248 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.735145    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.770124 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.770089    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 21:57:37.773112 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.773082    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8b73054c-openshift-default-dockercfg-z9v7n\""
Nov 01 21:57:37.836625 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.836587    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.836964 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.836935    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837114    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837322 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837296    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837532 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837482    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837671 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837646    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvlw\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837813 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837733    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837813 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837760    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.837813 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.837790    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.838302 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.838280    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.838952 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.838931    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.839683 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.839652    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.840295 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.840264    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.841451 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.841429    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.842240 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.842212    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.844040 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.844017    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.897900 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.897867    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.898074 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.897938    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvlw\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\") pod \"e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") " pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:37.983063 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:37.983018    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:38.416644 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:57:38.416612    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9f3934_f629_4cee_84f9_72e6a663ec73.slice/crio-beec146530d0d1ca7ffea6f54af3aec2e11f7b25420d253e50c90c662d16bbb1 WatchSource:0}: Error finding container beec146530d0d1ca7ffea6f54af3aec2e11f7b25420d253e50c90c662d16bbb1: Status 404 returned error can't find the container with id beec146530d0d1ca7ffea6f54af3aec2e11f7b25420d253e50c90c662d16bbb1
Nov 01 21:57:38.416994 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.416970    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 21:57:38.418759 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.418730    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:38.418845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.418789    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:38.418845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.418816    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:38.508977 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.508908    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerStarted","Data":"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"}
Nov 01 21:57:38.508977 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.508969    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerStarted","Data":"beec146530d0d1ca7ffea6f54af3aec2e11f7b25420d253e50c90c662d16bbb1"}
Nov 01 21:57:38.630735 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.630666    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podStartSLOduration=1.6306489050000001 podStartE2EDuration="1.630648905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:38.628775322 +0000 UTC m=+6912.013672914" watchObservedRunningTime="2026-11-01 21:57:38.630648905 +0000 UTC m=+6912.015546508"
Nov 01 21:57:38.983602 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:38.983562    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:39.988818 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:39.988781    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:40.519983 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:40.519938    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:40.520893 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:40.520867    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:57:45.095359 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.095319    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:57:45.100584 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.100554    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.156021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.155990    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:57:45.220139 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.219859    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8b73054c\""
Nov 01 21:57:45.220461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.220436    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8b73054c-dockercfg-tnzjp\""
Nov 01 21:57:45.229187 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.229148    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6svx\" (UniqueName: \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.229595 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.229571    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.274646 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.274614    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:57:45.278357 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.278337    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.313121 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.313079    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:57:45.330870 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.330839    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtcc\" (UniqueName: \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.331061 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.330960    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.331131 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.331066    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6svx\" (UniqueName: \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.331194 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.331156    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.331930 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.331909    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.389419 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.389343    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6svx\" (UniqueName: \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\") pod \"payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.415606 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.415570    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:45.432236 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.432205    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtcc\" (UniqueName: \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.432352 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.432285    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.433232 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.433205    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.494969 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.494932    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtcc\" (UniqueName: \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\") pod \"payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") " pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.591207 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.591165    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:45.798134 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.798100    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:57:45.798984 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:57:45.798959    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73be0941_d6d3_48cd_b924_695e910d76e5.slice/crio-f7cef5e41d437584c536c6c56eaef50af8fc6796df1e757117430a357e02eb4c WatchSource:0}: Error finding container f7cef5e41d437584c536c6c56eaef50af8fc6796df1e757117430a357e02eb4c: Status 404 returned error can't find the container with id f7cef5e41d437584c536c6c56eaef50af8fc6796df1e757117430a357e02eb4c
Nov 01 21:57:45.978143 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:57:45.978113    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a61ee22_2934_4278_82ef_be0075625286.slice/crio-9dccf167e3ba9189ac07fe56724047b7d25eeb4b8789227b8ead716e480ae9e2 WatchSource:0}: Error finding container 9dccf167e3ba9189ac07fe56724047b7d25eeb4b8789227b8ead716e480ae9e2: Status 404 returned error can't find the container with id 9dccf167e3ba9189ac07fe56724047b7d25eeb4b8789227b8ead716e480ae9e2
Nov 01 21:57:45.978747 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:45.978718    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:57:46.554875 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.554841    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" event={"ID":"3a61ee22-2934-4278-82ef-be0075625286","Type":"ContainerStarted","Data":"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"}
Nov 01 21:57:46.555398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.554904    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" event={"ID":"3a61ee22-2934-4278-82ef-be0075625286","Type":"ContainerStarted","Data":"9dccf167e3ba9189ac07fe56724047b7d25eeb4b8789227b8ead716e480ae9e2"}
Nov 01 21:57:46.555398 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.555025    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:46.556470 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.556438    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" event={"ID":"73be0941-d6d3-48cd-b924-695e910d76e5","Type":"ContainerStarted","Data":"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"}
Nov 01 21:57:46.556597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.556508    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:57:46.556597 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.556534    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" event={"ID":"73be0941-d6d3-48cd-b924-695e910d76e5","Type":"ContainerStarted","Data":"f7cef5e41d437584c536c6c56eaef50af8fc6796df1e757117430a357e02eb4c"}
Nov 01 21:57:46.670356 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.670306    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" podStartSLOduration=1.6702916079999999 podStartE2EDuration="1.670291608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:46.667903437 +0000 UTC m=+6920.052801030" watchObservedRunningTime="2026-11-01 21:57:46.670291608 +0000 UTC m=+6920.055189200"
Nov 01 21:57:46.789674 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:46.785674    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" podStartSLOduration=1.785653372 podStartE2EDuration="1.785653372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:46.784127316 +0000 UTC m=+6920.169024913" watchObservedRunningTime="2026-11-01 21:57:46.785653372 +0000 UTC m=+6920.170550964"
Nov 01 21:57:47.563105 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.563073    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:57:47.566811 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.566794    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.625533 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.619929    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8b73054c-serving-cert\""
Nov 01 21:57:47.625533 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.625346    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:57:47.656204 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.656171    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.656409 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.656357    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.656513 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.656471    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7z7\" (UniqueName: \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.656918 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.656893    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.758522 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.758462    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.758736 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.758567    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.758736 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.758595    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7z7\" (UniqueName: \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.758736 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:57:47.758613    5066 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-serving-cert: secret "maas-api-e2e-mt-8b73054c-serving-cert" not found
Nov 01 21:57:47.758736 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.758658    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.758736 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:57:47.758686    5066 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls podName:708aca3e-512c-4880-95f4-27fcd8a3ef52 nodeName:}" failed. No retries permitted until 2026-11-01 21:57:48.258666019 +0000 UTC m=+6921.643563609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls") pod "maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" (UID: "708aca3e-512c-4880-95f4-27fcd8a3ef52") : secret "maas-api-e2e-mt-8b73054c-serving-cert" not found
Nov 01 21:57:47.759284 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.759260    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.761212 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.761187    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:47.822248 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:47.822158    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7z7\" (UniqueName: \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:48.264783 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:48.264745    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:48.267307 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:48.267284    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") pod \"maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:48.480379 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:48.480343    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:48.855923 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:48.855892    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:57:48.856336 ip-10-0-142-223 kubenswrapper[5066]: W1101 21:57:48.856308    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708aca3e_512c_4880_95f4_27fcd8a3ef52.slice/crio-dcd9923f33a7b8bd847d38808ef20b8057c85a000aa93eaf7608aa855bb7da7b WatchSource:0}: Error finding container dcd9923f33a7b8bd847d38808ef20b8057c85a000aa93eaf7608aa855bb7da7b: Status 404 returned error can't find the container with id dcd9923f33a7b8bd847d38808ef20b8057c85a000aa93eaf7608aa855bb7da7b
Nov 01 21:57:48.857941 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:48.857918    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:57:49.578430 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:49.578390    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" event={"ID":"708aca3e-512c-4880-95f4-27fcd8a3ef52","Type":"ContainerStarted","Data":"dcd9923f33a7b8bd847d38808ef20b8057c85a000aa93eaf7608aa855bb7da7b"}
Nov 01 21:57:50.596792 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:50.596743    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" event={"ID":"708aca3e-512c-4880-95f4-27fcd8a3ef52","Type":"ContainerStarted","Data":"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"}
Nov 01 21:57:50.597197 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:50.596842    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:50.717185 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:50.717126    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" podStartSLOduration=2.8006628940000002 podStartE2EDuration="3.717110285s" totalImagesPullingTime="916.447391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:50.71414375 +0000 UTC m=+6924.099041341" watchObservedRunningTime="2026-11-01 21:57:50.717110285 +0000 UTC m=+6924.102007878"
Nov 01 21:57:51.628689 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:57:51.628655    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:57:56.610022 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:56.609984    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:57:57.585765 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:57.585721    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:57:57.585979 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:57:57.585838    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:58:05.718845 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.718810    5066 generic.go:356] "Generic (PLEG): container finished" podID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerID="4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f" exitCode=0
Nov 01 21:58:05.719297 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.718873    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerDied","Data":"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"}
Nov 01 21:58:05.719297 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.719264    5066 scope.go:122] "RemoveContainer" containerID="4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"
Nov 01 21:58:05.720699 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.720660    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:05.720809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.720739    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:05.720809 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:05.720769    5066 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:06.729869 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:06.729827    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerStarted","Data":"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"}
Nov 01 21:58:06.985787 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:06.985684    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:58:06.986830 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:06.986798    5066 patch_prober.go:29] interesting pod/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 21:58:06.986960 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:06.986851    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 21:58:07.985929 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:07.985891    5066 patch_prober.go:29] interesting pod/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 21:58:07.986371 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:07.985950    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 21:58:08.985956 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:08.985908    5066 patch_prober.go:29] interesting pod/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 21:58:08.986358 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:08.985983    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 21:58:09.985964 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:09.985911    5066 patch_prober.go:29] interesting pod/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 21:58:09.986460 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:09.985983    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 21:58:10.520546 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:10.520483    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:58:10.985920 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:10.985875    5066 patch_prober.go:29] interesting pod/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 21:58:10.986103 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:10.985953    5066 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 21:58:11.987632 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:11.987599    5066 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:58:11.988706 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:58:11.988686    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 21:58:21.629811 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:58:21.629766    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:58:51.628675 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:58:51.628588    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:59:21.629078 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:59:21.629039    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:59:43.858228 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:43.858192    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:59:43.965825 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:43.965790    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:59:48.859540 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:48.859484    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" podUID="3a61ee22-2934-4278-82ef-be0075625286" containerName="payload-processing" containerID="cri-o://a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102" gracePeriod=25
Nov 01 21:59:48.967303 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:48.967240    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" podUID="73be0941-d6d3-48cd-b924-695e910d76e5" containerName="payload-pre-processing" containerID="cri-o://1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773" gracePeriod=25
Nov 01 21:59:49.118417 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.118350    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:59:49.231793 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.231764    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:59:49.260338 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.260303    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\") pod \"3a61ee22-2934-4278-82ef-be0075625286\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") "
Nov 01 21:59:49.260553 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.260418    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\" (UniqueName: \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\") pod \"3a61ee22-2934-4278-82ef-be0075625286\" (UID: \"3a61ee22-2934-4278-82ef-be0075625286\") "
Nov 01 21:59:49.260728 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.260699    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume" pod "3a61ee22-2934-4278-82ef-be0075625286" (UID: "3a61ee22-2934-4278-82ef-be0075625286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:49.260929 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.260911    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a61ee22-2934-4278-82ef-be0075625286-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.262856 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.262832    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc" pod "3a61ee22-2934-4278-82ef-be0075625286" (UID: "3a61ee22-2934-4278-82ef-be0075625286"). InnerVolumeSpecName "kube-api-access-9wtcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:49.323591 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.323560    5066 generic.go:356] "Generic (PLEG): container finished" podID="73be0941-d6d3-48cd-b924-695e910d76e5" containerID="1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773" exitCode=0
Nov 01 21:59:49.323776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.323618    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"
Nov 01 21:59:49.323776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.323644    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" event={"ID":"73be0941-d6d3-48cd-b924-695e910d76e5","Type":"ContainerDied","Data":"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"}
Nov 01 21:59:49.323776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.323695    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv" event={"ID":"73be0941-d6d3-48cd-b924-695e910d76e5","Type":"ContainerDied","Data":"f7cef5e41d437584c536c6c56eaef50af8fc6796df1e757117430a357e02eb4c"}
Nov 01 21:59:49.323776 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.323716    5066 scope.go:122] "RemoveContainer" containerID="1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"
Nov 01 21:59:49.325211 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.325192    5066 generic.go:356] "Generic (PLEG): container finished" podID="3a61ee22-2934-4278-82ef-be0075625286" containerID="a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102" exitCode=0
Nov 01 21:59:49.325298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.325249    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" event={"ID":"3a61ee22-2934-4278-82ef-be0075625286","Type":"ContainerDied","Data":"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"}
Nov 01 21:59:49.325298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.325252    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"
Nov 01 21:59:49.325298 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.325277    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7" event={"ID":"3a61ee22-2934-4278-82ef-be0075625286","Type":"ContainerDied","Data":"9dccf167e3ba9189ac07fe56724047b7d25eeb4b8789227b8ead716e480ae9e2"}
Nov 01 21:59:49.334413 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.334273    5066 scope.go:122] "RemoveContainer" containerID="1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"
Nov 01 21:59:49.334722 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:59:49.334694    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773\": container with ID starting with 1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773 not found: ID does not exist" containerID="1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"
Nov 01 21:59:49.334795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.334731    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773"} err="failed to get container status \"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773\": rpc error: code = NotFound desc = could not find container \"1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773\": container with ID starting with 1dd8ac4cf60871b224121ddbd46f90bb2e5c31b4e77eb5429440e8045b75b773 not found: ID does not exist"
Nov 01 21:59:49.334795 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.334748    5066 scope.go:122] "RemoveContainer" containerID="a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"
Nov 01 21:59:49.343962 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.343948    5066 scope.go:122] "RemoveContainer" containerID="a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"
Nov 01 21:59:49.344231 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:59:49.344211    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102\": container with ID starting with a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102 not found: ID does not exist" containerID="a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"
Nov 01 21:59:49.344283 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.344238    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102"} err="failed to get container status \"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102\": rpc error: code = NotFound desc = could not find container \"a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102\": container with ID starting with a460e4eeaf769e67e2bb88106fcc807531098da705ef6f46ba80a42021dea102 not found: ID does not exist"
Nov 01 21:59:49.361692 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.361664    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\" (UniqueName: \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\") pod \"73be0941-d6d3-48cd-b924-695e910d76e5\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") "
Nov 01 21:59:49.361860 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.361748    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\") pod \"73be0941-d6d3-48cd-b924-695e910d76e5\" (UID: \"73be0941-d6d3-48cd-b924-695e910d76e5\") "
Nov 01 21:59:49.362138 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.362114    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume" pod "73be0941-d6d3-48cd-b924-695e910d76e5" (UID: "73be0941-d6d3-48cd-b924-695e910d76e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:49.362550 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.362528    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtcc\" (UniqueName: \"kubernetes.io/projected/3a61ee22-2934-4278-82ef-be0075625286-kube-api-access-9wtcc\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.362649 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.362554    5066 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be0941-d6d3-48cd-b924-695e910d76e5-plugins-config-volume\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.364456 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.364431    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx" pod "73be0941-d6d3-48cd-b924-695e910d76e5" (UID: "73be0941-d6d3-48cd-b924-695e910d76e5"). InnerVolumeSpecName "kube-api-access-t6svx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:49.463852 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.463774    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:59:49.464021 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.463944    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6svx\" (UniqueName: \"kubernetes.io/projected/73be0941-d6d3-48cd-b924-695e910d76e5-kube-api-access-t6svx\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.467424 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.467397    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b73054c-547f65486c-fqsx7"]
Nov 01 21:59:49.746655 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.746620    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:59:49.750129 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:49.750107    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b73054c-79847776c9-7gvhv"]
Nov 01 21:59:50.035863 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.035762    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:59:50.036343 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.036036    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" podUID="708aca3e-512c-4880-95f4-27fcd8a3ef52" containerName="maas-api" containerID="cri-o://8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93" gracePeriod=30
Nov 01 21:59:50.311039 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.311011    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:59:50.332040 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.332006    5066 generic.go:356] "Generic (PLEG): container finished" podID="708aca3e-512c-4880-95f4-27fcd8a3ef52" containerID="8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93" exitCode=0
Nov 01 21:59:50.332244 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.332051    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"
Nov 01 21:59:50.332244 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.332087    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" event={"ID":"708aca3e-512c-4880-95f4-27fcd8a3ef52","Type":"ContainerDied","Data":"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"}
Nov 01 21:59:50.332244 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.332139    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc" event={"ID":"708aca3e-512c-4880-95f4-27fcd8a3ef52","Type":"ContainerDied","Data":"dcd9923f33a7b8bd847d38808ef20b8057c85a000aa93eaf7608aa855bb7da7b"}
Nov 01 21:59:50.332244 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.332165    5066 scope.go:122] "RemoveContainer" containerID="8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"
Nov 01 21:59:50.341874 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.341851    5066 scope.go:122] "RemoveContainer" containerID="8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"
Nov 01 21:59:50.342276 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:59:50.342251    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93\": container with ID starting with 8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93 not found: ID does not exist" containerID="8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"
Nov 01 21:59:50.342388 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.342286    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93"} err="failed to get container status \"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93\": rpc error: code = NotFound desc = could not find container \"8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93\": container with ID starting with 8335d688cd582da380700559540ecd559fc6d9e757c9a8bb429a1443bd07ea93 not found: ID does not exist"
Nov 01 21:59:50.476617 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.476577    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\") pod \"708aca3e-512c-4880-95f4-27fcd8a3ef52\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") "
Nov 01 21:59:50.476879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.476685    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") pod \"708aca3e-512c-4880-95f4-27fcd8a3ef52\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") "
Nov 01 21:59:50.476879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.476722    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\") pod \"708aca3e-512c-4880-95f4-27fcd8a3ef52\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") "
Nov 01 21:59:50.476879 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.476777    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\" (UniqueName: \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\") pod \"708aca3e-512c-4880-95f4-27fcd8a3ef52\" (UID: \"708aca3e-512c-4880-95f4-27fcd8a3ef52\") "
Nov 01 21:59:50.477036 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.476940    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca" pod "708aca3e-512c-4880-95f4-27fcd8a3ef52" (UID: "708aca3e-512c-4880-95f4-27fcd8a3ef52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:50.477154 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.477138    5066 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708aca3e-512c-4880-95f4-27fcd8a3ef52-openshift-service-ca\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:50.478936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.478910    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls" pod "708aca3e-512c-4880-95f4-27fcd8a3ef52" (UID: "708aca3e-512c-4880-95f4-27fcd8a3ef52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:50.478936 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.478930    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs" pod "708aca3e-512c-4880-95f4-27fcd8a3ef52" (UID: "708aca3e-512c-4880-95f4-27fcd8a3ef52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:50.479147 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.478998    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7" pod "708aca3e-512c-4880-95f4-27fcd8a3ef52" (UID: "708aca3e-512c-4880-95f4-27fcd8a3ef52"). InnerVolumeSpecName "kube-api-access-2b7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:50.578219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.578137    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7z7\" (UniqueName: \"kubernetes.io/projected/708aca3e-512c-4880-95f4-27fcd8a3ef52-kube-api-access-2b7z7\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:50.578219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.578172    5066 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-maas-api-tls\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:50.578219 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.578182    5066 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708aca3e-512c-4880-95f4-27fcd8a3ef52-metrics-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 21:59:50.731454 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.731413    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a61ee22-2934-4278-82ef-be0075625286" path="/var/lib/kubelet/pods/3a61ee22-2934-4278-82ef-be0075625286/volumes"
Nov 01 21:59:50.731989 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.731966    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73be0941-d6d3-48cd-b924-695e910d76e5" path="/var/lib/kubelet/pods/73be0941-d6d3-48cd-b924-695e910d76e5/volumes"
Nov 01 21:59:50.755461 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.755419    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:59:50.760815 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:50.760784    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8b73054c-644859d9cd-mg6kc"]
Nov 01 21:59:51.633559 ip-10-0-142-223 kubenswrapper[5066]: E1101 21:59:51.633522    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 21:59:52.730144 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:52.730099    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708aca3e-512c-4880-95f4-27fcd8a3ef52" path="/var/lib/kubelet/pods/708aca3e-512c-4880-95f4-27fcd8a3ef52/volumes"
Nov 01 21:59:57.706628 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:57.706586    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 21:59:57.707073 ip-10-0-142-223 kubenswrapper[5066]: I1101 21:59:57.706807    5066 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy" containerID="cri-o://2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0" gracePeriod=30
Nov 01 22:00:02.981851 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:02.981787    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 22:00:03.003917 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.003885    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004093 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004037    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004093 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004083    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004198 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004125    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004198 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004153    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004198 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004193    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004364 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004239    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004364 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004278    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004471 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004361    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:03.004471 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004380    5066 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\") pod \"9a9f3934-f629-4cee-84f9-72e6a663ec73\" (UID: \"9a9f3934-f629-4cee-84f9-72e6a663ec73\") "
Nov 01 22:00:03.004641 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004547    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:03.004641 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004565    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:03.004737 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004597    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:00:03.004797 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004657    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:03.004928 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004907    5066 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-credential-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.004996 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004933    5066 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-socket\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.004996 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004948    5066 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-workload-certs\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.004996 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004963    5066 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a9f3934-f629-4cee-84f9-72e6a663ec73-istiod-ca-cert\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.004996 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.004977    5066 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-data\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.006570 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.006532    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:03.006685 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.006564    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:03.006685 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.006643    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "kube-api-access-cjvlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:03.006685 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.006651    5066 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo" pod "9a9f3934-f629-4cee-84f9-72e6a663ec73" (UID: "9a9f3934-f629-4cee-84f9-72e6a663ec73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:00:03.105729 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.105693    5066 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-token\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.105729 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.105724    5066 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-envoy\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.105729 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.105736    5066 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a9f3934-f629-4cee-84f9-72e6a663ec73-istio-podinfo\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.105977 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.105747    5066 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvlw\" (UniqueName: \"kubernetes.io/projected/9a9f3934-f629-4cee-84f9-72e6a663ec73-kube-api-access-cjvlw\") on node \"ip-10-0-142-223.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 22:00:03.434828 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.434726    5066 generic.go:356] "Generic (PLEG): container finished" podID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerID="2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0" exitCode=0
Nov 01 22:00:03.434828 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.434781    5066 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"
Nov 01 22:00:03.434828 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.434814    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerDied","Data":"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"}
Nov 01 22:00:03.435098 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.434872    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs" event={"ID":"9a9f3934-f629-4cee-84f9-72e6a663ec73","Type":"ContainerDied","Data":"beec146530d0d1ca7ffea6f54af3aec2e11f7b25420d253e50c90c662d16bbb1"}
Nov 01 22:00:03.435098 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.434898    5066 scope.go:122] "RemoveContainer" containerID="2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"
Nov 01 22:00:03.444962 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.444944    5066 scope.go:122] "RemoveContainer" containerID="4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"
Nov 01 22:00:03.453739 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.453712    5066 scope.go:122] "RemoveContainer" containerID="2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"
Nov 01 22:00:03.454018 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:00:03.453996    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0\": container with ID starting with 2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0 not found: ID does not exist" containerID="2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"
Nov 01 22:00:03.454065 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.454027    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0"} err="failed to get container status \"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0\": rpc error: code = NotFound desc = could not find container \"2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0\": container with ID starting with 2b89245eafe58b1be18cf1a95eb113a0c7bffef3bbf1f3d7a9d38600506641e0 not found: ID does not exist"
Nov 01 22:00:03.454065 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.454045    5066 scope.go:122] "RemoveContainer" containerID="4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"
Nov 01 22:00:03.454267 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:00:03.454249    5066 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f\": container with ID starting with 4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f not found: ID does not exist" containerID="4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"
Nov 01 22:00:03.454317 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.454274    5066 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f"} err="failed to get container status \"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f\": rpc error: code = NotFound desc = could not find container \"4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f\": container with ID starting with 4b56a7f9e567348d2e8726c1b82773496457330870de206bd99d6e8ad4bb511f not found: ID does not exist"
Nov 01 22:00:03.607613 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.607579    5066 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 22:00:03.610866 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:03.610842    5066 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8b73054c-openshift-default-bb48998bf-7bzcs"]
Nov 01 22:00:04.730151 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:00:04.730114    5066 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" path="/var/lib/kubelet/pods/9a9f3934-f629-4cee-84f9-72e6a663ec73/volumes"
Nov 01 22:00:21.629593 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:00:21.629481    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:00:51.629724 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:00:51.629683    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:01:18.545519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.544624    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:01:18.545519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.544729    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:01:18.545519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.545193    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:01:18.545519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.545251    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:01:18.586665 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.586615    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:01:18.586964 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.586932    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:01:18.994705 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.994648    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:01:18.995088 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:01:18.995053    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:01:21.628707 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:01:21.628668    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:01:51.643595 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:01:51.643551    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:02:21.629194 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:02:21.629147    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:02:27.776175 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:02:27.776144    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 22:02:27.855941 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:02:27.855913    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 22:02:51.772035 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:02:51.771841    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:03:03.546518 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.545610    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:03:03.546518 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.545704    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:03:03.546518 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.546207    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:03:03.546518 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.546262    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:03:03.588529 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.587623    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:03:03.588529 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.587712    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:03:03.981674 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.981624    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:03:03.982046 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:03:03.982011    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:03:21.629297 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:03:21.629214    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:03:51.630402 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:03:51.630364    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:04:21.629738 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:04:21.629698    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:04:51.740663 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:04:51.740413    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:05:21.629215 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:05:21.629175    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:05:33.541519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.540893    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-p6j5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:05:33.541519 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.541004    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-p6j5v" podUID="e7e5f891-7a78-440a-ad1f-2eedf6aad509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:05:33.542295 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.541728    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qr2z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:05:33.542295 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.541788    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qr2z2" podUID="61a6346f-9976-47c6-9006-8fbfdc4f07e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:05:33.583516 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.582725    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pnwrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:05:33.583516 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.582817    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pnwrg" podUID="19ecb3f1-43e1-44dd-b904-055ce99077b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:05:33.979655 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.979608    5066 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7lgkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:05:33.980004 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:05:33.979967    5066 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7lgkh" podUID="2459e512-5a19-46b0-967d-881f764b0c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:05:51.629611 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:05:51.629571    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:06:21.630219 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:06:21.630134    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:06:51.628967 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:06:51.628929    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:06:54.260483 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:06:54.260440    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cmbpq_be0efee3-c422-4d59-a0b7-67c656df942c/manager/0.log"
Nov 01 22:06:57.537023 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:06:57.536992    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8595f64f4d-kz76k_e693c822-8f7c-4b73-b122-dba7095bca4c/maas-api/0.log"
Nov 01 22:07:03.517542 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:03.517476    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4pll5_631dad46-02be-4fba-a072-43be25b7a0e5/manager/0.log"
Nov 01 22:07:04.892850 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:04.892802    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mfmm4_6b6e656f-8138-4f53-835d-f3fa7b8bc9d2/manager/0.log"
Nov 01 22:07:06.691647 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:06.691618    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv_fb53d414-7ae6-4124-be91-0552575a3cfd/istio-proxy/0.log"
Nov 01 22:07:07.917685 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:07.917653    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-7lgkh_2459e512-5a19-46b0-967d-881f764b0c8d/istio-proxy/0.log"
Nov 01 22:07:09.061407 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:09.061366    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-p6j5v_e7e5f891-7a78-440a-ad1f-2eedf6aad509/istio-proxy/0.log"
Nov 01 22:07:09.319475 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:09.319400    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-pnwrg_19ecb3f1-43e1-44dd-b904-055ce99077b7/istio-proxy/0.log"
Nov 01 22:07:09.530323 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:09.530288    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-qr2z2_61a6346f-9976-47c6-9006-8fbfdc4f07e5/istio-proxy/0.log"
Nov 01 22:07:09.995873 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:09.995838    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv_5bc1fcc8-7c32-487d-b2d0-5b0645c43a26/istio-proxy/0.log"
Nov 01 22:07:10.199889 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:10.199852    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d98d48688-6pcws_0164ec25-f653-42a1-9860-8470c532ee23/payload-pre-processing/0.log"
Nov 01 22:07:10.418510 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:10.418391    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7f5ccd4859-xftxz_939548ca-a52e-46d5-ae33-f57e79fefd64/payload-processing/0.log"
Nov 01 22:07:10.638665 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:10.638633    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-646fc8f47-znjf4_1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/router/0.log"
Nov 01 22:07:11.676914 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:11.676880    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-r95bc_0feecfe1-8475-436d-ad16-ac1f80c1208e/main/0.log"
Nov 01 22:07:12.648147 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:12.648118    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8p8b9n_e0c79ef7-cb8d-454a-ad7a-04a878f241d4/main/0.log"
Nov 01 22:07:13.078256 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:13.078227    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-wjzjv_9caa90c5-0265-43fa-b45d-6d52778e11df/main/0.log"
Nov 01 22:07:13.289191 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:13.289159    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-46tcv_2171233f-acc1-4ada-84ad-bfdfa7d4c834/main/0.log"
Nov 01 22:07:21.629348 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:07:21.629309    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:07:23.355856 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.355821    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h9v88/must-gather-fbpdl"]
Nov 01 22:07:23.356528 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356473    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.356528 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356513    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356538    5066 cpu_manager.go:414] "containerMap: removing container" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356548    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356562    5066 cpu_manager.go:414] "containerMap: removing container" podUID="3a61ee22-2934-4278-82ef-be0075625286" containerName="payload-processing"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356570    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a61ee22-2934-4278-82ef-be0075625286" containerName="payload-processing"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356604    5066 cpu_manager.go:414] "containerMap: removing container" podUID="708aca3e-512c-4880-95f4-27fcd8a3ef52" containerName="maas-api"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356613    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708aca3e-512c-4880-95f4-27fcd8a3ef52" containerName="maas-api"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356651    5066 cpu_manager.go:414] "containerMap: removing container" podUID="73be0941-d6d3-48cd-b924-695e910d76e5" containerName="payload-pre-processing"
Nov 01 22:07:23.356703 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356659    5066 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73be0941-d6d3-48cd-b924-695e910d76e5" containerName="payload-pre-processing"
Nov 01 22:07:23.357116 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356794    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="708aca3e-512c-4880-95f4-27fcd8a3ef52" containerName="maas-api"
Nov 01 22:07:23.357116 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356813    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="73be0941-d6d3-48cd-b924-695e910d76e5" containerName="payload-pre-processing"
Nov 01 22:07:23.357116 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356826    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.357116 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356836    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9f3934-f629-4cee-84f9-72e6a663ec73" containerName="istio-proxy"
Nov 01 22:07:23.357116 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.356847    5066 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a61ee22-2934-4278-82ef-be0075625286" containerName="payload-processing"
Nov 01 22:07:23.360783 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.360760    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.397313 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.397276    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8m8h\" (UniqueName: \"kubernetes.io/projected/239cd4c0-f506-4f7d-9367-a8b097af29dc-kube-api-access-s8m8h\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.397574 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.397523    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239cd4c0-f506-4f7d-9367-a8b097af29dc-must-gather-output\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.415705 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.415668    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/must-gather-fbpdl"]
Nov 01 22:07:23.462361 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.462326    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"kube-root-ca.crt\""
Nov 01 22:07:23.462571 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.462386    5066 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"openshift-service-ca.crt\""
Nov 01 22:07:23.462571 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.462391    5066 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h9v88\"/\"default-dockercfg-rbhbt\""
Nov 01 22:07:23.498311 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.498276    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8m8h\" (UniqueName: \"kubernetes.io/projected/239cd4c0-f506-4f7d-9367-a8b097af29dc-kube-api-access-s8m8h\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.498522 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.498417    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239cd4c0-f506-4f7d-9367-a8b097af29dc-must-gather-output\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.498796 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.498777    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/239cd4c0-f506-4f7d-9367-a8b097af29dc-must-gather-output\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.555815 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.555777    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8m8h\" (UniqueName: \"kubernetes.io/projected/239cd4c0-f506-4f7d-9367-a8b097af29dc-kube-api-access-s8m8h\") pod \"must-gather-fbpdl\" (UID: \"239cd4c0-f506-4f7d-9367-a8b097af29dc\") " pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:23.676179 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:23.676097    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/must-gather-fbpdl"
Nov 01 22:07:24.052428 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:24.052333    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/must-gather-fbpdl"]
Nov 01 22:07:24.052675 ip-10-0-142-223 kubenswrapper[5066]: W1101 22:07:24.052648    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod239cd4c0_f506_4f7d_9367_a8b097af29dc.slice/crio-4346fddd3c2ece491865eeb473c105b583fa485bdf1c4fa90c6e8569f64dc551 WatchSource:0}: Error finding container 4346fddd3c2ece491865eeb473c105b583fa485bdf1c4fa90c6e8569f64dc551: Status 404 returned error can't find the container with id 4346fddd3c2ece491865eeb473c105b583fa485bdf1c4fa90c6e8569f64dc551
Nov 01 22:07:24.054593 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:24.054574    5066 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:07:24.337063 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:24.336974    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/must-gather-fbpdl" event={"ID":"239cd4c0-f506-4f7d-9367-a8b097af29dc","Type":"ContainerStarted","Data":"4346fddd3c2ece491865eeb473c105b583fa485bdf1c4fa90c6e8569f64dc551"}
Nov 01 22:07:26.352146 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:26.352103    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/must-gather-fbpdl" event={"ID":"239cd4c0-f506-4f7d-9367-a8b097af29dc","Type":"ContainerStarted","Data":"fc4b2404bd1d139af7a89d03734fda2ebb0269cc6b83eddeb031b7095654ecf4"}
Nov 01 22:07:26.352594 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:26.352155    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/must-gather-fbpdl" event={"ID":"239cd4c0-f506-4f7d-9367-a8b097af29dc","Type":"ContainerStarted","Data":"bf09e499d8297d0f10139a0d0c66c2346b35e1909d00a4ae9bb95edc3f31b96b"}
Nov 01 22:07:26.468343 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:26.468284    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h9v88/must-gather-fbpdl" podStartSLOduration=2.204053321 podStartE2EDuration="3.468265141s" totalImagesPullingTime="1.26421182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:26.465053521 +0000 UTC m=+7499.849951111" watchObservedRunningTime="2026-11-01 22:07:26.468265141 +0000 UTC m=+7499.853162773"
Nov 01 22:07:27.813429 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:27.813398    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 22:07:27.899349 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:27.899322    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dlrcw_ce41e940-ffc1-4f08-93c8-3abd565d89e2/console-operator/2.log"
Nov 01 22:07:32.936944 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:32.936900    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nn89g_2a563d52-ad66-4e70-98e9-10266df43ddd/global-pull-secret-syncer/0.log"
Nov 01 22:07:34.112133 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:34.112100    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2zq5k_befdc8ba-b69c-4357-a5c8-fb748a4cb03e/connection-checker/0.log"
Nov 01 22:07:34.547514 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:34.547342    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ckjjq_07b921bf-2def-4eaf-8efd-a883c81f402a/connection-checker/0.log"
Nov 01 22:07:34.981146 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:34.981115    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qgkq7_395408fa-e36c-4283-82ee-613469a4af49/connection-checker/0.log"
Nov 01 22:07:35.548835 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:35.545783    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cmbpq_be0efee3-c422-4d59-a0b7-67c656df942c/manager/0.log"
Nov 01 22:07:36.305329 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:36.305292    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ktldr_8398821f-a5b8-4ddf-a46f-e1123e5bfbfc/konnectivity-agent/0.log"
Nov 01 22:07:37.544523 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:37.544473    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-223.us-west-1.compute.internal_88d138da692bc0e2c453d9d2e3886bfc/haproxy/0.log"
Nov 01 22:07:41.147310 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:41.147262    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cmbpq_be0efee3-c422-4d59-a0b7-67c656df942c/manager/0.log"
Nov 01 22:07:42.589054 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.589017    5066 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"]
Nov 01 22:07:42.597152 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.597118    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.653029 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.652982    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"]
Nov 01 22:07:42.740626 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.740594    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-proc\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.740796 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.740657    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-podres\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.740796 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.740716    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-sys\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.740796 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.740738    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-lib-modules\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.740796 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.740794    5066 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslld\" (UniqueName: \"kubernetes.io/projected/28183ed0-5d7e-4efd-9ad8-136882a2ad00-kube-api-access-xslld\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842122 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.841980    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-sys\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842122 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842043    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-lib-modules\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842403 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842142    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslld\" (UniqueName: \"kubernetes.io/projected/28183ed0-5d7e-4efd-9ad8-136882a2ad00-kube-api-access-xslld\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842403 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842086    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-sys\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842403 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842221    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-lib-modules\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842403 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842290    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-proc\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842666 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842413    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-proc\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842666 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842407    5066 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-podres\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.842666 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.842487    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28183ed0-5d7e-4efd-9ad8-136882a2ad00-podres\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.900303 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.900265    5066 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslld\" (UniqueName: \"kubernetes.io/projected/28183ed0-5d7e-4efd-9ad8-136882a2ad00-kube-api-access-xslld\") pod \"perf-node-gather-daemonset-9h4mq\" (UID: \"28183ed0-5d7e-4efd-9ad8-136882a2ad00\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:42.917694 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:42.917660    5066 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:43.088904 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.088859    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4pll5_631dad46-02be-4fba-a072-43be25b7a0e5/manager/0.log"
Nov 01 22:07:43.296571 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.296540    5066 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"]
Nov 01 22:07:43.297328 ip-10-0-142-223 kubenswrapper[5066]: W1101 22:07:43.297299    5066 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28183ed0_5d7e_4efd_9ad8_136882a2ad00.slice/crio-30c8aa4a39a41c669c77c372f6451756413f4568532454e6c03d333eee8285ff WatchSource:0}: Error finding container 30c8aa4a39a41c669c77c372f6451756413f4568532454e6c03d333eee8285ff: Status 404 returned error can't find the container with id 30c8aa4a39a41c669c77c372f6451756413f4568532454e6c03d333eee8285ff
Nov 01 22:07:43.507585 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.507540    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq" event={"ID":"28183ed0-5d7e-4efd-9ad8-136882a2ad00","Type":"ContainerStarted","Data":"28c6d87047667160ded136067ef438a3cb2ab16a3f67386247fc19e469073b5d"}
Nov 01 22:07:43.507804 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.507609    5066 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq" event={"ID":"28183ed0-5d7e-4efd-9ad8-136882a2ad00","Type":"ContainerStarted","Data":"30c8aa4a39a41c669c77c372f6451756413f4568532454e6c03d333eee8285ff"}
Nov 01 22:07:43.507804 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.507725    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:43.622620 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:43.622511    5066 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq" podStartSLOduration=1.622481678 podStartE2EDuration="1.622481678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:43.620461607 +0000 UTC m=+7517.005359199" watchObservedRunningTime="2026-11-01 22:07:43.622481678 +0000 UTC m=+7517.007379269"
Nov 01 22:07:44.603434 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:44.603396    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mfmm4_6b6e656f-8138-4f53-835d-f3fa7b8bc9d2/manager/0.log"
Nov 01 22:07:46.359981 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:46.359943    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cmbpq_be0efee3-c422-4d59-a0b7-67c656df942c/manager/0.log"
Nov 01 22:07:49.525871 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:49.525841    5066 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-9h4mq"
Nov 01 22:07:51.081744 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:51.081706    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-5nd9r_0a4a52df-9a7b-4332-92ec-8ddc8d299bad/cluster-monitoring-operator/0.log"
Nov 01 22:07:51.630703 ip-10-0-142-223 kubenswrapper[5066]: E1101 22:07:51.630658    5066 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sr5gb" containerName="extractor"
Nov 01 22:07:52.146596 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:52.146565    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv_fb53d414-7ae6-4124-be91-0552575a3cfd/istio-proxy/0.log"
Nov 01 22:07:55.637935 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:55.637899    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-7lgkh_2459e512-5a19-46b0-967d-881f764b0c8d/istio-proxy/0.log"
Nov 01 22:07:56.081933 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:56.081900    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2bkv_642b34b2-e573-4b35-b98f-24c98279ccf5/node-exporter/0.log"
Nov 01 22:07:56.495791 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:56.495756    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2bkv_642b34b2-e573-4b35-b98f-24c98279ccf5/kube-rbac-proxy/0.log"
Nov 01 22:07:56.781020 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:56.780898    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-p6j5v_e7e5f891-7a78-440a-ad1f-2eedf6aad509/istio-proxy/0.log"
Nov 01 22:07:56.912343 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:56.912316    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2bkv_642b34b2-e573-4b35-b98f-24c98279ccf5/init-textfile/0.log"
Nov 01 22:07:56.989858 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:56.989828    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-pnwrg_19ecb3f1-43e1-44dd-b904-055ce99077b7/istio-proxy/0.log"
Nov 01 22:07:57.199967 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:57.199878    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-qr2z2_61a6346f-9976-47c6-9006-8fbfdc4f07e5/istio-proxy/0.log"
Nov 01 22:07:57.623202 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:57.623163    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-nn2hv_5bc1fcc8-7c32-487d-b2d0-5b0645c43a26/istio-proxy/0.log"
Nov 01 22:07:57.832483 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:57.832444    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7d98d48688-6pcws_0164ec25-f653-42a1-9860-8470c532ee23/payload-pre-processing/0.log"
Nov 01 22:07:58.043758 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:58.043702    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7f5ccd4859-xftxz_939548ca-a52e-46d5-ae33-f57e79fefd64/payload-processing/0.log"
Nov 01 22:07:58.261024 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:07:58.260986    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-646fc8f47-znjf4_1e92c0a9-d5ef-49ff-905f-9dbdb07d5ad6/router/0.log"
Nov 01 22:08:02.355724 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:08:02.355692    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lg5b7_689090e6-11bf-4c1d-8d64-9d38d2ab888a/prometheus-operator-admission-webhook/0.log"
Nov 01 22:08:02.959128 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:08:02.959097    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4pll5_631dad46-02be-4fba-a072-43be25b7a0e5/manager/0.log"
Nov 01 22:08:04.035916 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:08:04.035867    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mfmm4_6b6e656f-8138-4f53-835d-f3fa7b8bc9d2/manager/0.log"
Nov 01 22:08:05.985013 ip-10-0-142-223 kubenswrapper[5066]: I1101 22:08:05.984973    5066 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfdvvfv_fb53d414-7ae6-4124-be91-0552575a3cfd/istio-proxy/0.log"